Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Improved RequestList docs (#573)

Jan Čurnf600875013cf
diff --git a/docs/typedefs/RequestListOptions.md b/docs/typedefs/RequestListOptions.md
index 5d273ede8..639848c70 100644
--- a/docs/typedefs/RequestListOptions.md
+++ b/docs/typedefs/RequestListOptions.md
@@ -24,13 +24,22 @@ title: RequestListOptions
which will instruct <code>RequestList</code> to download the source URLs from a given remote location.
The URLs will be parsed from the received response.</p>
<pre><code>[
- // One URL
+ // A single URL
{ method: &#39;GET&#39;, url: &#39;http://example.com/a/b&#39; },
- // Batch import of URLs from a file hosted on the web
+
+ // Batch import of URLs from a file hosted on the web,
+ // where the URLs should be requested using the HTTP POST request
{ method: &#39;POST&#39;, requestsFromUrl: &#39;http://example.com/urls.txt&#39; },
- // Batch import combined with regex.
- { method: &#39;POST&#39;, requestsFromUrl: &#39;http://example.com/urls.txt&#39;, regex: /https:\/\/example.com\/.+/ },
+
+ // Batch import from remote file, using a specific regular expression to extract the URLs.
+ { requestsFromUrl: &#39;http://example.com/urls.txt&#39;, regex: /https:\/\/example.com\/.+/ },
+
+ // Get list of URLs from a Google Sheets document. Just add &quot;/gviz/tq?tqx=out:csv&quot; to the Google Sheet URL.
+ // For details, see https://help.apify.com/en/articles/2906022-scraping-a-list-of-urls-from-a-google-sheets-document
+ { requestsFromUrl: &#39;https://docs.google.com/spreadsheets/d/1GA5sSQhQjB_REes8I5IKg31S-TuRcznWOPjcpNqtxmU/gviz/tq?tqx=out:csv&#39; }
+
]</code></pre></td></tr><tr>
+
<td><code>[persistStateKey]</code></td><td><code>String</code></td><td></td>
</tr>
<tr>
diff --git a/src/request_list.js b/src/request_list.js
index fb4a4e98a..98cf9600a 100644
--- a/src/request_list.js
+++ b/src/request_list.js
@@ -21,12 +21,19 @@ export const SOURCES_PERSISTENCE_KEY = 'REQUEST_LIST_SOURCES';
*
* ```
* [
- * // One URL
+ * // A single URL
* { method: 'GET', url: 'http://example.com/a/b' },
- * // Batch import of URLs from a file hosted on the web
+ *
+ * // Batch import of URLs from a file hosted on the web,
+ * // where the URLs should be requested using the HTTP POST request
* { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt' },
- * // Batch import combined with regex.
- * { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt', regex: /https:\/\/example.com\/.+/ },
+ *
+ * // Batch import from remote file, using a specific regular expression to extract the URLs.
+ * { requestsFromUrl: 'http://example.com/urls.txt', regex: /https:\/\/example.com\/.+/ },
+ *
+ * // Get list of URLs from a Google Sheets document. Just add "/gviz/tq?tqx=out:csv" to the Google Sheet URL.
+ * // For details, see https://help.apify.com/en/articles/2906022-scraping-a-list-of-urls-from-a-google-sheets-document
+ * { requestsFromUrl: 'https://docs.google.com/spreadsheets/d/1GA5sSQhQjB_REes8I5IKg31S-TuRcznWOPjcpNqtxmU/gviz/tq?tqx=out:csv' }
* ]
* ```
* @property {String} [persistStateKey]
diff --git a/types/request_list.d.ts b/types/request_list.d.ts
index ae971dfa5..1b9d56803 100644
--- a/types/request_list.d.ts
+++ b/types/request_list.d.ts
@@ -11,12 +11,19 @@ export const SOURCES_PERSISTENCE_KEY: "REQUEST_LIST_SOURCES";
*
* ```
* [
- * // One URL
+ * // A single URL
* { method: 'GET', url: 'http://example.com/a/b' },
- * // Batch import of URLs from a file hosted on the web
+ *
+ * // Batch import of URLs from a file hosted on the web,
+ * // where the URLs should be requested using the HTTP POST request
* { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt' },
- * // Batch import combined with regex.
- * { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt', regex: /https:\/\/example.com\/.+/ },
+ *
+ * // Batch import from remote file, using a specific regular expression to extract the URLs.
+ * { requestsFromUrl: 'http://example.com/urls.txt', regex: /https:\/\/example.com\/.+/ },
+ *
+ * // Get list of URLs from a Google Sheets document. Just add "/gviz/tq?tqx=out:csv" to the Google Sheet URL.
+ * // For details, see https://help.apify.com/en/articles/2906022-scraping-a-list-of-urls-from-a-google-sheets-document
+ * { requestsFromUrl: 'https://docs.google.com/spreadsheets/d/1GA5sSQhQjB_REes8I5IKg31S-TuRcznWOPjcpNqtxmU/gviz/tq?tqx=out:csv' }
* ]
* ```
* @property {String} [persistStateKey]
@@ -305,12 +312,19 @@ export type RequestListOptions = {
*
* ```
* [
- * // One URL
+ * // A single URL
* { method: 'GET', url: 'http://example.com/a/b' },
- * // Batch import of URLs from a file hosted on the web
+ *
+ * // Batch import of URLs from a file hosted on the web,
+ * // where the URLs should be requested using the HTTP POST request
* { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt' },
- * // Batch import combined with regex.
- * { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt', regex: /https:\/\/example.com\/.+/ },
+ *
+ * // Batch import from remote file, using a specific regular expression to extract the URLs.
+ * { requestsFromUrl: 'http://example.com/urls.txt', regex: /https:\/\/example.com\/.+/ },
+ *
+ * // Get list of URLs from a Google Sheets document. Just add "/gviz/tq?tqx=out:csv" to the Google Sheet URL.
+ * // For details, see https://help.apify.com/en/articles/2906022-scraping-a-list-of-urls-from-a-google-sheets-document
+ * { requestsFromUrl: 'https://docs.google.com/spreadsheets/d/1GA5sSQhQjB_REes8I5IKg31S-TuRcznWOPjcpNqtxmU/gviz/tq?tqx=out:csv' }
* ]
* ```
*/