Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Updated docs

Jan Curne60a6d54fcda
diff --git a/src/request_list.js b/src/request_list.js
index 38c9a5ec6..4135fd9bc 100644
--- a/src/request_list.js
+++ b/src/request_list.js
@@ -23,7 +23,13 @@ const ensureUniqueKeyValid = (uniqueKey) => {
/**
* Represents a static list of URLs to crawl.
* The URLs can be provided either in code or parsed from a text file hosted on the web.
+ *
* Each URL is represented using an instance of the {@link Request|`Request`} class.
+ * The list can only contain unique URLs. More precisely, it can only contain `Request` instances
+ * with distinct `uniqueKey` properties. By default, `uniqueKey` is generated from the URL, but it can also be overridden.
+ * To add a single URL multiple times to the list,
+ * corresponding `Request` objects will need to have different `uniqueKey` properties.
+ * You can use the `keepDuplicateUrls` option to do this for you.
*
* Once you create an instance of `RequestList`, you need to call {@link RequestList#initialize|`initialize()`}
* before the instance can be used. After that, no more URLs can be added to the list.
@@ -110,10 +116,10 @@ const ensureUniqueKeyValid = (uniqueKey) => {
*
* @param {Boolean} [options.keepDuplicateUrls=false]
* By default, `RequestList` will deduplicate the provided URLs. Default deduplication is based
- * on the `uniqueKey` property of passed sources. If not present, it is generated by normalizing
- * the URL. If present, it is kept intact. In any case, only one request per `uniqueKey` is added
+ * on the `uniqueKey` property of passed source {@link Request} objects. If the property is not present,
+ * it is generated by normalizing the URL. If present, it is kept intact. In any case, only one request per `uniqueKey` is added
* to the `RequestList` resulting in removing of duplicate URLs / unique keys.
- * Setting `keepDuplicateUrls` to true will append an additional identifier to the `uniqueKey`
+ * Setting `keepDuplicateUrls` to `true` will append an additional identifier to the `uniqueKey`
* of each request that does not already include a `uniqueKey`. Therefore, duplicate
* URLs will be kept in the list. It does not protect the user from having duplicates in user set
* `uniqueKey`s however. It is the user's responsibility to ensure uniqueness of their unique keys,
diff --git a/src/request_queue.js b/src/request_queue.js
index f7646ad8b..5280c3731 100644
--- a/src/request_queue.js
+++ b/src/request_queue.js
@@ -120,7 +120,12 @@ const getRequestId = (uniqueKey) => {
* Represents a queue of URLs to crawl, which is used for deep crawling of websites
* where you start with several URLs and then recursively
* follow links to other pages. The data structure supports both breadth-first and depth-first crawling orders.
- * Each URL is represented as an instance of the {@link Request|`Request`} class.
+ *
+ * Each URL is represented using an instance of the {@link Request|`Request`} class.
+ * The queue can only contain unique URLs. More precisely, it can only contain `Request` instances
+ * with distinct `uniqueKey` properties. By default, `uniqueKey` is generated from the URL, but it can also be overridden.
+ * To add a single URL multiple times to the queue,
+ * corresponding `Request` objects will need to have different `uniqueKey` properties.
*
* Do not instantiate this class directly, use the
* {@link Apify#openRequestQueue|`Apify.openRequestQueue()`} function instead.