Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Better help for RequestList

Jan Curne3400a3cb174
diff --git a/src/request_list.js b/src/request_list.js
index f9241f069..7ec0a43c8 100644
--- a/src/request_list.js
+++ b/src/request_list.js
@@ -28,8 +28,8 @@ const ensureUniqueKeyValid = (uniqueKey) => {
};
/**
- * Pprovides way to handle a list of URLs to be crawled.
- * Each URL is reprented using an instance of the `Request` class.
+ * Provides way to handle a list of URLs to be crawled.
+ * Each URL is represented using an instance of the `Request` class.
*
* `RequestList` has an internal state where it remembers handled requests, requests in progress and also reclaimed requests.
* The state might be persisted in a key-value store as shown in the example below so if an act is restarted (due to internal
@@ -48,6 +48,7 @@ const ensureUniqueKeyValid = (uniqueKey) => {
* // Note that all URLs must start with http:// or https://
* { requestsFromUrl: 'http://www.example.com/my-url-list.txt', userData: { isFromUrl: true } },
* ],
+ * persistStateKey: 'my-crawling-state'
* });
*
* await requestList.initialize(); // Load requests.
@@ -74,7 +75,11 @@ const ensureUniqueKeyValid = (uniqueKey) => {
* { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt' },
* ]
* ```
- * @param {Object} [options.state] State of the `RequestList` to be initialized from. It is in the form returned by `requestList.getState()`:
+ * @param {String} [options.persistStateKey] Key-value store key under which the `RequestList` persists its state. If this is set then `RequestList`
+ * persists its state in regular intervals and loads the state from there in a case that's restarted
+ * due to some error or migration to another worker machine.
+ * @param {Object} [options.state] The state object that the `RequestList` will be initialized from.
+ * It is in the form returned by `requestList.getState()`, such as follows:
* ```javascript
* {
* nextIndex: 5,
@@ -85,9 +90,7 @@ const ensureUniqueKeyValid = (uniqueKey) => {
* },
* }
* ```
- * @param {String} [options.persistStateKey] Key-value store key under which RequestList persists its state. If this is set then RequestList
- * persists its state in regular intervals and loads the state from there in a case thats restarted
- * due to some error or migration to another worker machine.
+ * Note that the preferred (and simpler) way to persist the state of crawling of the `RequestList` is to use the `persistStateKey` parameter instead.
*/
export default class RequestList {
constructor(opts = {}) {