Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Fixed docs for utils.downloadListOfUrls()

Jan Curn899efe65d312
diff --git a/src/utils.js b/src/utils.js
index 65572a567..9b6cc137b 100644
--- a/src/utils.js
+++ b/src/utils.js
@@ -359,9 +359,11 @@ const sleep = (millis) => {
* Optionally, custom regular expression and encoding may be provided.
*
* @param {Object} options
- * @param {String} options.url
- * @param {String} [options.encoding='utf8']
+ * @param {String} options.url URL to the file
+ * @param {String} [options.encoding='utf8'] The encoding of the file.
* @param {RegExp} [options.urlRegExp=URL_NO_COMMAS_REGEX]
+ * Custom regular expression to identify the URLs in the file to extract.
+ * The regular expression should be case-insensitive and have global flag set (i.e. `/something/gi`).
* @returns {Promise<String[]>}
* @memberOf utils
*/