Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Improved docs
Jan Curn9e1fa362b668
diff --git a/src/request_list.js b/src/request_list.jsindex 0e2c2893e..b2aa51f4a 100644--- a/src/request_list.js+++ b/src/request_list.js@@ -76,10 +76,10 @@ export const SOURCES_PERSISTENCE_KEY = 'REQUEST_LIST_SOURCES';* @param {Object} options All `RequestList` parameters are passed* via an options object with the following keys:* @param {Array} options.sources- * An array of sources for the `RequestList`. Its contents can either be just plain objects,- * defining at least the 'url' property or instances of the {@link Request} class.- * Additionally a `requestsFromUrl` property may be used instead of `url`,- * which will instruct the `RequestList` to download the sources from the given remote location.+ * An array of sources of URLs for the `RequestList`. It can be either an array of plain objects that+ * define the `url` property, or an array of instances of the {@link Request} class.+ * Additionally, the `requestsFromUrl` property may be used instead of `url`,+ * which will instruct `RequestList` to download the source URLs from a given remote location.* The URLs will be parsed from the received response.** ```@@ -604,10 +604,15 @@ export class RequestList {* If `null`, the list will not be persisted and will only be stored in memory. Process restart* will then cause the list to be crawled again from the beginning. We suggest always using a name.* @param {Object[]|string[]} sources- * Sources represent the URLs to crawl. It can either be a `string[]` with plain URLs or an `Object[]`.- * The objects' contents can either be just plain objects, defining at least the 'url' property- * or instances of the {@link Request} class. See the (`new RequestList`)(RequestList#new_RequestList_new)- * options for details.+ * An array of sources of URLs for the `RequestList`.+ * It can be either an array of plain objects that+ * define the `url` property, or an array of instances of the {@link Request} class.+ * Additionally, the `requestsFromUrl` property may be used instead of `url`,+ * which will instruct `RequestList` to download the source URLs from a given remote location.+ * The URLs will be parsed from the received response.+ *+ * For details, see the (`RequestList`)[requestlist#new-exportsrequestlistoptions]+ * constructor options.* @param {Object} [options]* The (`new RequestList`)(RequestList#new_RequestList_new) options. Note that the listName parameter supersedes* the `persistStateKey` and `persistSourcesKey` options and the sources parameter supersedes the `sources` option.