Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Improved crawler docs

Jan Curn0dc1e19b79f4
diff --git a/src/basic_crawler.js b/src/basic_crawler.js
index cf978e29d..b014d476f 100644
--- a/src/basic_crawler.js
+++ b/src/basic_crawler.js
@@ -33,9 +33,14 @@ const DEFAULT_OPTIONS = {
};
/**
- * Provides a simple framework for parallel crawling of web pages,
- * whose URLs are fed either from a static list
- * or from a dynamic queue of URLs.
+ * Provides a simple framework for parallel crawling of web pages.
+ * The URLs to crawl are fed either from a static list of URLs
+ * or from a dynamic queue of URLs enabling recursive crawling of websites.
+ *
+ * `BasicCrawler` is a low-level tool that requires the user to implement the page
+ * download and data extraction functionality themselves.
+ * If you want a crawler that already facilitates this functionality,
+ * please consider using {@link PuppeteerCrawler} or {@link CheerioCrawler}.
*
* `BasicCrawler` invokes the user-provided [`handleRequestFunction()`](#new_BasicCrawler_new)
* for each {@link Request} object, which represents a single URL to crawl.
diff --git a/src/cheerio_crawler.js b/src/cheerio_crawler.js
index 81483395d..e5d7555af 100644
--- a/src/cheerio_crawler.js
+++ b/src/cheerio_crawler.js
@@ -38,6 +38,13 @@ const DEFAULT_OPTIONS = {
/**
* Provides a framework for the parallel crawling of web pages using plain HTTP requests and
* <a href="https://www.npmjs.com/package/cheerio" target="_blank">cheerio</a> HTML parser.
+ * The URLs to crawl are fed either from a static list of URLs
+ * or from a dynamic queue of URLs enabling recursive crawling of websites.
+ *
+ * Since `CheerioCrawler` uses raw HTTP requests to download web pages,
+ * it is very fast and efficient on data bandwidth. However, if the target website requires JavaScript
+ * to display the content, you might need to use {@link PuppeteerCrawler} instead,
+ * because it loads the pages using full-featured headless Chrome browser.
*
* `CheerioCrawler` downloads each URL using a plain HTTP request,
* parses the HTML content using <a href="https://www.npmjs.com/package/cheerio" target="_blank">Cheerio</a>
diff --git a/src/puppeteer_crawler.js b/src/puppeteer_crawler.js
index 4f6d6c26a..47560efa7 100644
--- a/src/puppeteer_crawler.js
+++ b/src/puppeteer_crawler.js
@@ -19,6 +19,13 @@ const DEFAULT_OPTIONS = {
/**
* Provides a simple framework for parallel crawling of web pages
* using headless Chrome with <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a>.
+ * The URLs to crawl are fed either from a static list of URLs
+ * or from a dynamic queue of URLs enabling recursive crawling of websites.
+ *
+ * Since `PuppeteerCrawler` uses headless Chrome to download web pages and extract data,
+ * it is useful for crawling of websites that require to execute JavaScript.
+ * If the target website doesn't need JavaScript, consider using {@link CheerioCrawler},
+ * which downloads the pages using raw HTTP requests and is about 10x faster.
*
* The source URLs are represented using {@link Request} objects that are fed from
* {@link RequestList} or {@link RequestQueue} instances provided by the [`requestList`](#new_PuppeteerCrawler_new)