Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Improved RequestList docs (#573)
Jan Čurnf600875013cf
diff --git a/docs/typedefs/RequestListOptions.md b/docs/typedefs/RequestListOptions.mdindex 5d273ede8..639848c70 100644--- a/docs/typedefs/RequestListOptions.md+++ b/docs/typedefs/RequestListOptions.md@@ -24,13 +24,22 @@ title: RequestListOptionswhich will instruct <code>RequestList</code> to download the source URLs from a given remote location.The URLs will be parsed from the received response.</p><pre><code>[- // One URL+ // A single URL{ method: 'GET', url: 'http://example.com/a/b' },- // Batch import of URLs from a file hosted on the web++ // Batch import of URLs from a file hosted on the web,+ // where the URLs should be requested using the HTTP POST request{ method: 'POST', requestsFromUrl: 'http://example.com/urls.txt' },- // Batch import combined with regex.- { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt', regex: /https:\/\/example.com\/.+/ },++ // Batch import from remote file, using a specific regular expression to extract the URLs.+ { requestsFromUrl: 'http://example.com/urls.txt', regex: /https:\/\/example.com\/.+/ },++ // Get list of URLs from a Google Sheets document. Just add "/gviz/tq?tqx=out:csv" to the Google Sheet URL.+ // For details, see https://help.apify.com/en/articles/2906022-scraping-a-list-of-urls-from-a-google-sheets-document+ { requestsFromUrl: 'https://docs.google.com/spreadsheets/d/1GA5sSQhQjB_REes8I5IKg31S-TuRcznWOPjcpNqtxmU/gviz/tq?tqx=out:csv' }+]</code></pre></td></tr><tr>+<td><code>[persistStateKey]</code></td><td><code>String</code></td><td></td></tr><tr>diff --git a/src/request_list.js b/src/request_list.jsindex fb4a4e98a..98cf9600a 100644--- a/src/request_list.js+++ b/src/request_list.js@@ -21,12 +21,19 @@ export const SOURCES_PERSISTENCE_KEY = 'REQUEST_LIST_SOURCES';** ```* [- * // One URL+ * // A single URL* { method: 'GET', url: 'http://example.com/a/b' },- * // Batch import of URLs from a file hosted on the web+ *+ * // Batch import of URLs from a file hosted on the web,+ * // where the URLs should be requested using the HTTP POST request* { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt' },- * // Batch import combined with regex.- * { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt', regex: /https:\/\/example.com\/.+/ },+ *+ * // Batch import from remote file, using a specific regular expression to extract the URLs.+ * { requestsFromUrl: 'http://example.com/urls.txt', regex: /https:\/\/example.com\/.+/ },+ *+ * // Get list of URLs from a Google Sheets document. Just add "/gviz/tq?tqx=out:csv" to the Google Sheet URL.+ * // For details, see https://help.apify.com/en/articles/2906022-scraping-a-list-of-urls-from-a-google-sheets-document+ * { requestsFromUrl: 'https://docs.google.com/spreadsheets/d/1GA5sSQhQjB_REes8I5IKg31S-TuRcznWOPjcpNqtxmU/gviz/tq?tqx=out:csv' }* ]* ```* @property {String} [persistStateKey]diff --git a/types/request_list.d.ts b/types/request_list.d.tsindex ae971dfa5..1b9d56803 100644--- a/types/request_list.d.ts+++ b/types/request_list.d.ts@@ -11,12 +11,19 @@ export const SOURCES_PERSISTENCE_KEY: "REQUEST_LIST_SOURCES";** ```* [- * // One URL+ * // A single URL* { method: 'GET', url: 'http://example.com/a/b' },- * // Batch import of URLs from a file hosted on the web+ *+ * // Batch import of URLs from a file hosted on the web,+ * // where the URLs should be requested using the HTTP POST request* { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt' },- * // Batch import combined with regex.- * { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt', regex: /https:\/\/example.com\/.+/ },+ *+ * // Batch import from remote file, using a specific regular expression to extract the URLs.+ * { requestsFromUrl: 'http://example.com/urls.txt', regex: /https:\/\/example.com\/.+/ },+ *+ * // Get list of URLs from a Google Sheets document. Just add "/gviz/tq?tqx=out:csv" to the Google Sheet URL.+ * // For details, see https://help.apify.com/en/articles/2906022-scraping-a-list-of-urls-from-a-google-sheets-document+ * { requestsFromUrl: 'https://docs.google.com/spreadsheets/d/1GA5sSQhQjB_REes8I5IKg31S-TuRcznWOPjcpNqtxmU/gviz/tq?tqx=out:csv' }* ]* ```* @property {String} [persistStateKey]@@ -305,12 +312,19 @@ export type RequestListOptions = {** ```* [- * // One URL+ * // A single URL* { method: 'GET', url: 'http://example.com/a/b' },- * // Batch import of URLs from a file hosted on the web+ *+ * // Batch import of URLs from a file hosted on the web,+ * // where the URLs should be requested using the HTTP POST request* { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt' },- * // Batch import combined with regex.- * { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt', regex: /https:\/\/example.com\/.+/ },+ *+ * // Batch import from remote file, using a specific regular expression to extract the URLs.+ * { requestsFromUrl: 'http://example.com/urls.txt', regex: /https:\/\/example.com\/.+/ },+ *+ * // Get list of URLs from a Google Sheets document. Just add "/gviz/tq?tqx=out:csv" to the Google Sheet URL.+ * // For details, see https://help.apify.com/en/articles/2906022-scraping-a-list-of-urls-from-a-google-sheets-document+ * { requestsFromUrl: 'https://docs.google.com/spreadsheets/d/1GA5sSQhQjB_REes8I5IKg31S-TuRcznWOPjcpNqtxmU/gviz/tq?tqx=out:csv' }* ]* ```*/