Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
chore: improve templates and upgrading guide
Martin Adámeke0f6e628596d
diff --git a/docs/upgrading/upgrading_v3.md b/docs/upgrading/upgrading_v3.mdindex 657fa8e21..bd7f43018 100644--- a/docs/upgrading/upgrading_v3.md+++ b/docs/upgrading/upgrading_v3.md@@ -20,18 +20,18 @@ Up until version 3 of `apify`, the package contained both scraping related toolsThe [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of several smaller packages, released separately under `@crawlee` namespace:-- `@crawlee/core`: the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes-- `@crawlee/basic`: exports `BasicCrawler`-- `@crawlee/http`: exports `HttpCrawler` (which is used for creating `@crawlee/jsdom` and `@crawlee/cheerio`)-- `@crawlee/jsdom`: exports `JSDOMCrawler`-- `@crawlee/cheerio`: exports `CheerioCrawler`-- `@crawlee/browser`: exports `BrowserCrawler` (which is used for creating `@crawlee/playwright` and `@crawlee/puppeteer`)-- `@crawlee/playwright`: exports `PlaywrightCrawler`-- `@crawlee/puppeteer`: exports `PuppeteerCrawler`-- `@crawlee/memory-storage`: `@apify/storage-local` alternative-- `@crawlee/browser-pool`: previously `browser-pool` package-- `@crawlee/utils`: utility methods-- `@crawlee/types`: holds TS interfaces mainly about the `StorageClient`+- [`@crawlee/core`](https://crawlee.dev/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes+- [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`+- [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`+- [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`+- [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`+- [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`+- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))+- [`@crawlee/browser`](https://crawlee.dev/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler))+- [`@crawlee/memory-storage`](https://crawlee.dev/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative+- [`@crawlee/browser-pool`](https://crawlee.dev/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package+- [`@crawlee/utils`](https://crawlee.dev/api/utils): utility methods+- [`@crawlee/types`](https://crawlee.dev/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/api/core/interface/StorageClient)### Installing Crawleediff --git a/packages/templates/manifest.json b/packages/templates/manifest.jsonindex 13c9fe987..2a1f81744 100644--- a/packages/templates/manifest.json+++ b/packages/templates/manifest.json@@ -30,7 +30,7 @@},{"name": "puppeteer-js",- "description": "PlaywrightCrawler template project [JavaScript]"+ "description": "PuppeteerCrawler template project [JavaScript]"}]}diff --git a/packages/templates/templates/cheerio-js/src/routes.js b/packages/templates/templates/cheerio-js/src/routes.jsindex 299ce7913..4e491815d 100644--- a/packages/templates/templates/cheerio-js/src/routes.js+++ b/packages/templates/templates/cheerio-js/src/routes.js@@ -5,7 +5,7 @@ export const router = createCheerioRouter();router.addDefaultHandler(async ({ enqueueLinks, log }) => {log.info(`enqueueing new URLs`);await enqueueLinks({- globs: ['https://crawlee.dev/*'],+ globs: ['https://crawlee.dev/**'],label: 'detail',});});diff --git a/packages/templates/templates/cheerio-ts/src/routes.ts b/packages/templates/templates/cheerio-ts/src/routes.tsindex 299ce7913..4e491815d 100644--- a/packages/templates/templates/cheerio-ts/src/routes.ts+++ b/packages/templates/templates/cheerio-ts/src/routes.ts@@ -5,7 +5,7 @@ export const router = createCheerioRouter();router.addDefaultHandler(async ({ enqueueLinks, log }) => {log.info(`enqueueing new URLs`);await enqueueLinks({- globs: ['https://crawlee.dev/*'],+ globs: ['https://crawlee.dev/**'],label: 'detail',});});diff --git a/packages/templates/templates/playwright-js/src/routes.js b/packages/templates/templates/playwright-js/src/routes.jsindex 1d758a8f7..54a62847c 100644--- a/packages/templates/templates/playwright-js/src/routes.js+++ b/packages/templates/templates/playwright-js/src/routes.js@@ -5,7 +5,7 @@ export const router = createPlaywrightRouter();router.addDefaultHandler(async ({ enqueueLinks, log }) => {log.info(`enqueueing new URLs`);await enqueueLinks({- globs: ['https://crawlee.dev/*'],+ globs: ['https://crawlee.dev/**'],label: 'detail',});});diff --git a/packages/templates/templates/playwright-ts/src/routes.ts b/packages/templates/templates/playwright-ts/src/routes.tsindex 1d758a8f7..54a62847c 100644--- a/packages/templates/templates/playwright-ts/src/routes.ts+++ b/packages/templates/templates/playwright-ts/src/routes.ts@@ -5,7 +5,7 @@ export const router = createPlaywrightRouter();router.addDefaultHandler(async ({ enqueueLinks, log }) => {log.info(`enqueueing new URLs`);await enqueueLinks({- globs: ['https://crawlee.dev/*'],+ globs: ['https://crawlee.dev/**'],label: 'detail',});});diff --git a/packages/templates/templates/puppeteer-js/src/routes.js b/packages/templates/templates/puppeteer-js/src/routes.jsindex 52f55358c..5ddfeed13 100644--- a/packages/templates/templates/puppeteer-js/src/routes.js+++ b/packages/templates/templates/puppeteer-js/src/routes.js@@ -5,7 +5,7 @@ export const router = createPuppeteerRouter();router.addDefaultHandler(async ({ enqueueLinks, log }) => {log.info(`enqueueing new URLs`);await enqueueLinks({- globs: ['https://crawlee.dev/*'],+ globs: ['https://crawlee.dev/**'],label: 'detail',});});diff --git a/packages/templates/templates/puppeteer-ts/src/routes.ts b/packages/templates/templates/puppeteer-ts/src/routes.tsindex 52f55358c..5ddfeed13 100644--- a/packages/templates/templates/puppeteer-ts/src/routes.ts+++ b/packages/templates/templates/puppeteer-ts/src/routes.ts@@ -5,7 +5,7 @@ export const router = createPuppeteerRouter();router.addDefaultHandler(async ({ enqueueLinks, log }) => {log.info(`enqueueing new URLs`);await enqueueLinks({- globs: ['https://crawlee.dev/*'],+ globs: ['https://crawlee.dev/**'],label: 'detail',});});diff --git a/website/versioned_docs/version-3.0/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.0/upgrading/upgrading_v3.mdindex 657fa8e21..bd7f43018 100644--- a/website/versioned_docs/version-3.0/upgrading/upgrading_v3.md+++ b/website/versioned_docs/version-3.0/upgrading/upgrading_v3.md@@ -20,18 +20,18 @@ Up until version 3 of `apify`, the package contained both scraping related toolsThe [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of several smaller packages, released separately under `@crawlee` namespace:-- `@crawlee/core`: the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes-- `@crawlee/basic`: exports `BasicCrawler`-- `@crawlee/http`: exports `HttpCrawler` (which is used for creating `@crawlee/jsdom` and `@crawlee/cheerio`)-- `@crawlee/jsdom`: exports `JSDOMCrawler`-- `@crawlee/cheerio`: exports `CheerioCrawler`-- `@crawlee/browser`: exports `BrowserCrawler` (which is used for creating `@crawlee/playwright` and `@crawlee/puppeteer`)-- `@crawlee/playwright`: exports `PlaywrightCrawler`-- `@crawlee/puppeteer`: exports `PuppeteerCrawler`-- `@crawlee/memory-storage`: `@apify/storage-local` alternative-- `@crawlee/browser-pool`: previously `browser-pool` package-- `@crawlee/utils`: utility methods-- `@crawlee/types`: holds TS interfaces mainly about the `StorageClient`+- [`@crawlee/core`](https://crawlee.dev/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes+- [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`+- [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`+- [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`+- [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`+- [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`+- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))+- [`@crawlee/browser`](https://crawlee.dev/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler))+- [`@crawlee/memory-storage`](https://crawlee.dev/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative+- [`@crawlee/browser-pool`](https://crawlee.dev/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package+- [`@crawlee/utils`](https://crawlee.dev/api/utils): utility methods+- [`@crawlee/types`](https://crawlee.dev/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/api/core/interface/StorageClient)### Installing Crawlee