Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

chore: improve templates and upgrading guide

Martin Adámeke0f6e628596d
diff --git a/docs/upgrading/upgrading_v3.md b/docs/upgrading/upgrading_v3.md
index 657fa8e21..bd7f43018 100644
--- a/docs/upgrading/upgrading_v3.md
+++ b/docs/upgrading/upgrading_v3.md
@@ -20,18 +20,18 @@ Up until version 3 of `apify`, the package contained both scraping related tools
The [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of several smaller packages, released separately under `@crawlee` namespace:
-- `@crawlee/core`: the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes
-- `@crawlee/basic`: exports `BasicCrawler`
-- `@crawlee/http`: exports `HttpCrawler` (which is used for creating `@crawlee/jsdom` and `@crawlee/cheerio`)
-- `@crawlee/jsdom`: exports `JSDOMCrawler`
-- `@crawlee/cheerio`: exports `CheerioCrawler`
-- `@crawlee/browser`: exports `BrowserCrawler` (which is used for creating `@crawlee/playwright` and `@crawlee/puppeteer`)
-- `@crawlee/playwright`: exports `PlaywrightCrawler`
-- `@crawlee/puppeteer`: exports `PuppeteerCrawler`
-- `@crawlee/memory-storage`: `@apify/storage-local` alternative
-- `@crawlee/browser-pool`: previously `browser-pool` package
-- `@crawlee/utils`: utility methods
-- `@crawlee/types`: holds TS interfaces mainly about the `StorageClient`
+- [`@crawlee/core`](https://crawlee.dev/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes
+- [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`
+- [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`
+- [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`
+- [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`
+- [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`
+- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))
+- [`@crawlee/browser`](https://crawlee.dev/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler))
+- [`@crawlee/memory-storage`](https://crawlee.dev/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative
+- [`@crawlee/browser-pool`](https://crawlee.dev/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package
+- [`@crawlee/utils`](https://crawlee.dev/api/utils): utility methods
+- [`@crawlee/types`](https://crawlee.dev/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/api/core/interface/StorageClient)
### Installing Crawlee
diff --git a/packages/templates/manifest.json b/packages/templates/manifest.json
index 13c9fe987..2a1f81744 100644
--- a/packages/templates/manifest.json
+++ b/packages/templates/manifest.json
@@ -30,7 +30,7 @@
},
{
"name": "puppeteer-js",
- "description": "PlaywrightCrawler template project [JavaScript]"
+ "description": "PuppeteerCrawler template project [JavaScript]"
}
]
}
diff --git a/packages/templates/templates/cheerio-js/src/routes.js b/packages/templates/templates/cheerio-js/src/routes.js
index 299ce7913..4e491815d 100644
--- a/packages/templates/templates/cheerio-js/src/routes.js
+++ b/packages/templates/templates/cheerio-js/src/routes.js
@@ -5,7 +5,7 @@ export const router = createCheerioRouter();
router.addDefaultHandler(async ({ enqueueLinks, log }) => {
log.info(`enqueueing new URLs`);
await enqueueLinks({
- globs: ['https://crawlee.dev/*'],
+ globs: ['https://crawlee.dev/**'],
label: 'detail',
});
});
diff --git a/packages/templates/templates/cheerio-ts/src/routes.ts b/packages/templates/templates/cheerio-ts/src/routes.ts
index 299ce7913..4e491815d 100644
--- a/packages/templates/templates/cheerio-ts/src/routes.ts
+++ b/packages/templates/templates/cheerio-ts/src/routes.ts
@@ -5,7 +5,7 @@ export const router = createCheerioRouter();
router.addDefaultHandler(async ({ enqueueLinks, log }) => {
log.info(`enqueueing new URLs`);
await enqueueLinks({
- globs: ['https://crawlee.dev/*'],
+ globs: ['https://crawlee.dev/**'],
label: 'detail',
});
});
diff --git a/packages/templates/templates/playwright-js/src/routes.js b/packages/templates/templates/playwright-js/src/routes.js
index 1d758a8f7..54a62847c 100644
--- a/packages/templates/templates/playwright-js/src/routes.js
+++ b/packages/templates/templates/playwright-js/src/routes.js
@@ -5,7 +5,7 @@ export const router = createPlaywrightRouter();
router.addDefaultHandler(async ({ enqueueLinks, log }) => {
log.info(`enqueueing new URLs`);
await enqueueLinks({
- globs: ['https://crawlee.dev/*'],
+ globs: ['https://crawlee.dev/**'],
label: 'detail',
});
});
diff --git a/packages/templates/templates/playwright-ts/src/routes.ts b/packages/templates/templates/playwright-ts/src/routes.ts
index 1d758a8f7..54a62847c 100644
--- a/packages/templates/templates/playwright-ts/src/routes.ts
+++ b/packages/templates/templates/playwright-ts/src/routes.ts
@@ -5,7 +5,7 @@ export const router = createPlaywrightRouter();
router.addDefaultHandler(async ({ enqueueLinks, log }) => {
log.info(`enqueueing new URLs`);
await enqueueLinks({
- globs: ['https://crawlee.dev/*'],
+ globs: ['https://crawlee.dev/**'],
label: 'detail',
});
});
diff --git a/packages/templates/templates/puppeteer-js/src/routes.js b/packages/templates/templates/puppeteer-js/src/routes.js
index 52f55358c..5ddfeed13 100644
--- a/packages/templates/templates/puppeteer-js/src/routes.js
+++ b/packages/templates/templates/puppeteer-js/src/routes.js
@@ -5,7 +5,7 @@ export const router = createPuppeteerRouter();
router.addDefaultHandler(async ({ enqueueLinks, log }) => {
log.info(`enqueueing new URLs`);
await enqueueLinks({
- globs: ['https://crawlee.dev/*'],
+ globs: ['https://crawlee.dev/**'],
label: 'detail',
});
});
diff --git a/packages/templates/templates/puppeteer-ts/src/routes.ts b/packages/templates/templates/puppeteer-ts/src/routes.ts
index 52f55358c..5ddfeed13 100644
--- a/packages/templates/templates/puppeteer-ts/src/routes.ts
+++ b/packages/templates/templates/puppeteer-ts/src/routes.ts
@@ -5,7 +5,7 @@ export const router = createPuppeteerRouter();
router.addDefaultHandler(async ({ enqueueLinks, log }) => {
log.info(`enqueueing new URLs`);
await enqueueLinks({
- globs: ['https://crawlee.dev/*'],
+ globs: ['https://crawlee.dev/**'],
label: 'detail',
});
});
diff --git a/website/versioned_docs/version-3.0/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.0/upgrading/upgrading_v3.md
index 657fa8e21..bd7f43018 100644
--- a/website/versioned_docs/version-3.0/upgrading/upgrading_v3.md
+++ b/website/versioned_docs/version-3.0/upgrading/upgrading_v3.md
@@ -20,18 +20,18 @@ Up until version 3 of `apify`, the package contained both scraping related tools
The [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of several smaller packages, released separately under `@crawlee` namespace:
-- `@crawlee/core`: the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes
-- `@crawlee/basic`: exports `BasicCrawler`
-- `@crawlee/http`: exports `HttpCrawler` (which is used for creating `@crawlee/jsdom` and `@crawlee/cheerio`)
-- `@crawlee/jsdom`: exports `JSDOMCrawler`
-- `@crawlee/cheerio`: exports `CheerioCrawler`
-- `@crawlee/browser`: exports `BrowserCrawler` (which is used for creating `@crawlee/playwright` and `@crawlee/puppeteer`)
-- `@crawlee/playwright`: exports `PlaywrightCrawler`
-- `@crawlee/puppeteer`: exports `PuppeteerCrawler`
-- `@crawlee/memory-storage`: `@apify/storage-local` alternative
-- `@crawlee/browser-pool`: previously `browser-pool` package
-- `@crawlee/utils`: utility methods
-- `@crawlee/types`: holds TS interfaces mainly about the `StorageClient`
+- [`@crawlee/core`](https://crawlee.dev/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes
+- [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`
+- [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`
+- [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`
+- [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`
+- [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`
+- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))
+- [`@crawlee/browser`](https://crawlee.dev/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler))
+- [`@crawlee/memory-storage`](https://crawlee.dev/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative
+- [`@crawlee/browser-pool`](https://crawlee.dev/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package
+- [`@crawlee/utils`](https://crawlee.dev/api/utils): utility methods
+- [`@crawlee/types`](https://crawlee.dev/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/api/core/interface/StorageClient)
### Installing Crawlee