Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Improved docs

Jan Curn5e6cab3664a2
diff --git a/src/basic_crawler.js b/src/basic_crawler.js
index 365293fb1..8b4854420 100644
--- a/src/basic_crawler.js
+++ b/src/basic_crawler.js
@@ -83,10 +83,10 @@ const DEFAULT_OPTIONS = {
* The function must return a promise.
* @param {RequestList} options.requestList
* Static list of URLs to be processed.
- * Either `requestList` or `requestQueue` option must be provided.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
* @param {RequestQueue} options.requestQueue
* Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
- * Either `requestList` or `requestQueue` option must be provided.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
* @param {Function} [options.handleFailedRequestFunction]
* Function that handles requests that failed more then `options.maxRequestRetries` times.
* See source code on
diff --git a/src/cheerio_crawler.js b/src/cheerio_crawler.js
index 9b73ba239..9ba17395e 100644
--- a/src/cheerio_crawler.js
+++ b/src/cheerio_crawler.js
@@ -104,10 +104,10 @@ const DEFAULT_OPTIONS = {
* If the function returns a promise, it is awaited.
* @param {RequestList} options.requestList
* Static list of URLs to be processed.
- * Either `requestList` or `requestQueue` option must be provided.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
* @param {RequestQueue} options.requestQueue
* Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
- * Either `requestList` or `requestQueue` option must be provided.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
* @param {Function} [options.requestFunction]
* Overrides the default function that performs the HTTP request to get the raw HTML needed for Cheerio.
* See source code on <a href="https://github.com/apifytech/apify-js/blob/master/src/cheerio_crawler.js#L264">GitHub</a> for default behavior.
diff --git a/src/puppeteer_crawler.js b/src/puppeteer_crawler.js
index fc7d9a3cd..aa2deb9cf 100644
--- a/src/puppeteer_crawler.js
+++ b/src/puppeteer_crawler.js
@@ -95,10 +95,10 @@ const PAGE_CLOSE_TIMEOUT_MILLIS = 30000;
* `puppeteerPool` is an instance of the {@link PuppeteerPool} used by this `PuppeteerCrawler`.
* @param {RequestList} options.requestList
* Static list of URLs to be processed.
- * Either `requestList` or `requestQueue` option must be provided.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
* @param {RequestQueue} options.requestQueue
* Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
- * Either `requestList` or `requestQueue` option must be provided.
+ * Either `requestList` or `requestQueue` option must be provided (or both).
* @param {Number} [options.handlePageTimeoutSecs=300]
* Timeout in which the function passed as `options.handlePageFunction` needs to finish, in seconds.
* @param {Function} [options.gotoFunction]