Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Document usage of request list and request queue combination (#883)
Andrey Bykov49ba203632b1
diff --git a/README.md b/README.mdindex 5fb93dd73..6000e7b2b 100644--- a/README.md+++ b/README.md@@ -136,8 +136,7 @@ When you run the example, you should see Apify SDK automating a Chrome browser.By default, Apify SDK stores data to `./apify_storage` in the current working directory. You can override this behavior by setting either the-`APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variable. For details, see [Environment variables](https://sdk.apify.com/docs/guides/environment-variables) and-[Data storage](https://sdk.apify.com/docs/guides/data-storage).+`APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variable. For details, see [Environment variables](https://sdk.apify.com/docs/guides/environment-variables), [Request storage](https://sdk.apify.com/docs/guides/request-storage) and [Result storage](https://sdk.apify.com/docs/guides/result-storage).### Local usage with Apify command-line interface (CLI)diff --git a/docs/guides/apify_platform.md b/docs/guides/apify_platform.mdindex 810bcbd15..a2c016dfe 100644--- a/docs/guides/apify_platform.md+++ b/docs/guides/apify_platform.md@@ -5,7 +5,7 @@ title: Apify PlatformApify is a [platform](https://apify.com) built to serve large scale and high performance web scrapingand automation needs. It provides easy access to [compute instances (Actors)](#what-is-an-actor),-[convenient storages](../guides/data-storage), [proxies](../guides/proxy-management),+convenient [request](../guides/request-storage) and [result](../guides/result-storage) storages, [proxies](../guides/proxy-management),[scheduling](https://docs.apify.com/scheduler), [webhooks](https://docs.apify.com/webhooks)and [more](https://docs.apify.com/), accessible through a [web interface](https://my.apify.com)or an [API](https://docs.apify.com/api).diff --git a/docs/guides/quick_start.md b/docs/guides/quick_start.mdindex 7aee2a0dc..2701c5084 100644--- a/docs/guides/quick_start.md+++ b/docs/guides/quick_start.md@@ -55,8 +55,7 @@ When you run the example, you should see Apify SDK automating a Chrome browser.By default, Apify SDK stores data to `./apify_storage` in the current working directory. You can override this behavior by setting either the-`APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variable. For details, see [Environment variables](../guides/environment-variables) and-[Data storage](../guides/data-storage).+`APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variable. For details, see [Environment variables](../guides/environment-variables), [Request storage](../guides/request-storage) and [Result storage](../guides/result-storage).## Local usage with Apify command-line interface (CLI)diff --git a/docs/guides/request_storage.md b/docs/guides/request_storage.mdnew file mode 100644index 000000000..e38299a42--- /dev/null+++ b/docs/guides/request_storage.md@@ -0,0 +1,131 @@+---+id: request-storage+title: Request Storage+---++The Apify SDK has several request storage types that are useful for specific tasks. The requests are stored either on local disk to a directory defined by the+`APIFY_LOCAL_STORAGE_DIR` environment variable, or on the [Apify platform](/docs/guides/apify-platform) under the user account identified by the API token defined by the `APIFY_TOKEN` environment variable. If neither of these variables is defined, by default Apify SDK sets `APIFY_LOCAL_STORAGE_DIR` to `./apify_storage` in the current working directory and prints a warning.++Typically, you will be developing the code on your local computer and thus set the `APIFY_LOCAL_STORAGE_DIR` environment variable. Once the code is ready, you will deploy it to the Apify platform, where it will automatically set the `APIFY_TOKEN` environment variable and thus use cloud storage. No code changes are needed.++**Related links**++- [Apify platform storage documentation](https://docs.apify.com/storage)+- [View storage in Apify app](https://my.apify.com/storage)+- [Request queues API reference](https://docs.apify.com/api/v2#/reference/request-queues)++## Request queue++The request queue is a storage of URLs to crawl. The queue is used for the deep crawling of websites, where you start with several URLs and then recursively follow links to other pages. The data structure supports both breadth-first and depth-first crawling orders.++Each actor run is associated with a **default request queue**, which is created exclusively for the actor run. Typically, it is used to store URLs to crawl in the specific actor run. Its usage is optional.++In Apify SDK, the request queue is represented by the [`RequestQueue`](/docs/api/request-queue) class.++In local configuration, the request queue is emulated by [@apify/storage-local](https://github.com/apify/apify-storage-local-js) NPM package and its data is stored in SQLite database in the directory specified by the `APIFY_LOCAL_STORAGE_DIR` environment variable as follows:++```+{APIFY_LOCAL_STORAGE_DIR}/request_queues/{QUEUE_ID}/db.sqlite+```++Note that `{QUEUE_ID}` is the name or ID of the request queue. The default queue has ID `default`, unless you override it by setting the `APIFY_DEFAULT_REQUEST_QUEUE_ID` environment variable.++The following code demonstrates basic operations of the request queue:++```javascript+// Open the default request queue associated with the actor run+const requestQueue = await Apify.openRequestQueue();+// Enqueue the initial request+await requestQueue.addRequest({ url: 'https://example.com' });++// The crawler will automatically process requests from the queue+const crawler = new Apify.CheerioCrawler({+ requestQueue,+ handlePageFunction: async ({ $, request }) => {+ // Add new request to the queue+ await requestQueue.addRequest({ url: 'https://example.com/new-page' });+ // Add links found on page to the queue+ await Apify.utils.enqueueLinks({ $, requestQueue });+ },+});+```++To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/docs/examples/puppeteer-crawler) example.++## Request list++The request list is not a storage per se - it represents the list of URLs to crawl that is stored in a run memory (or optionally in default [Key-Value Store](../guides/results-storage#key-value-store) associated with the run, if specified). The list is used for the crawling of a large number of URLs, when you know all the URLs which should be visited by the crawler and no URLs would be added during the run. The URLs can be provided either in code or parsed from a text file hosted on the web.++Request list is created exclusively for the actor run and only if its usage is explicitly specified in the code. Its usage is optional.++In Apify SDK, the request list is represented by the [`RequestList`](/docs/api/request-list) class.++The following code demonstrates basic operations of the request list:++```javascript+// Prepare the sources array with URLs to visit+const sources = [+ { url: 'http://www.example.com/page-1' },+ { url: 'http://www.example.com/page-2' },+ { url: 'http://www.example.com/page-3' },+];+// Open the request list.+// List name is used to persist the sources and the list state in the key-value store+const requestList = await Apify.openRequestList('my-list', sources);++// The crawler will automatically process requests from the list+const crawler = new Apify.PuppeteerCrawler({+ requestList,+ handlePageFunction: async ({ page, request }) => {+ // Process the page (extract data, take page screenshot, etc).+ // No more requests could be added to the request list here+ },+});+```++To see more detailed example of how to use the request list with a crawler, see the [Puppeteer with proxy](/docs/examples/puppeteer-with-proxy) example.++## Which one to choose?++When using Request queue - you would normally have several start URLs (e.g. category pages on e-commerce website) and then recursively add more (e.g. individual item pages) programmatically to the queue, it supports dynamic adding and removing of requests. No more URLs can be added to Request list after its initialization as it is immutable, URLs cannot be removed from the list either.++On the other hand, the Request queue is not optimized for adding or removing numerous URLs in a batch. This is technically possible, but requests are added one by one to the queue, and thus it would take significant time with a larger number of requests. Request list however can contain even millions of URLs, and it would take significantly less time to add them to the list, compared to the queue.++Note that Request queue and Request list can be used together by the same crawler.+In such cases, each request from the Request list is enqueued into the Request queue first (to the foremost position in the queue, even if Request queue is not empty) and then consumed from the latter.+This is necessary to avoid the same URL being processed more than once (from the list first and then possibly from the queue).+In practical terms, such a combination can be useful when there are numerous initial URLs, but more URLs would be added dynamically by the crawler.++The following code demonstrates how to use Request queue and Request list in the same crawler:+```javascript+// Prepare the sources array with URLs to visit (it can contain millions of URLs)+const sources = [+ { url: 'http://www.example.com/page-1' },+ { url: 'http://www.example.com/page-2' },+ { url: 'http://www.example.com/page-3' },+];+// Open the request list+const requestList = await Apify.openRequestList('my-list', sources);++// Open the default request queue. It's not necessary to add any requests to the queue+const requestQueue = await Apify.openRequestQueue();++// The crawler will automatically process requests from the list and the queue+const crawler = new Apify.PuppeteerCrawler({+ requestList,+ requestQueue,+ // Each request from the request list is enqueued to the request queue one by one.+ // At this point request with the same URL would exist in the list and the queue+ handlePageFunction: async ({ request, page }) => {+ // Add new request to the queue+ await requestQueue.addRequest({ url: 'http://www.example.com/new-page' });++ // Add links found on page to the queue+ await Apify.utils.enqueueLinks({ page, requestQueue });++ // The requests above would be added to the queue (but not to the list)+ // and would be processed after the request list is empty.+ // No more requests could be added to the list here+ },+});+```diff --git a/docs/guides/data_storage.md b/docs/guides/result_storage.mdsimilarity index 69%rename from docs/guides/data_storage.mdrename to docs/guides/result_storage.mdindex 4d55b71bc..e168f3414 100644--- a/docs/guides/data_storage.md+++ b/docs/guides/result_storage.md@@ -1,9 +1,9 @@----id: data-storage-title: Data Storage+id: result-storage+title: Result Storage----The Apify SDK has several data storage types that are useful for specific tasks. The data is stored either on local disk to a directory defined by the+The Apify SDK has several result storage types that are useful for specific tasks. The data is stored either on local disk to a directory defined by the`APIFY_LOCAL_STORAGE_DIR` environment variable, or on the [Apify platform](../guides/apify-platform) under the user accountidentified by the API token defined by the `APIFY_TOKEN` environment variable. If neither of these variables is defined, by default Apify SDK sets`APIFY_LOCAL_STORAGE_DIR` to `./apify_storage` in the current working directory and prints a warning.@@ -16,7 +16,8 @@ code changes are needed.- [Apify platform storage documentation](https://docs.apify.com/storage)- [View storage in Apify app](https://my.apify.com/storage)-- [API reference](https://apify.com/docs/api/v2#/reference/key-value-stores)+- [Key-value stores API reference](https://apify.com/docs/api/v2#/reference/key-value-stores)+- [Datasets API reference](https://docs.apify.com/api/v2#/reference/datasets)## Key-value store@@ -112,51 +113,3 @@ await dataset.pushData([{ foo: 'bar2', col2: 'val2' }, { col3: 123 }]);```To see how to use the dataset to store crawler results, see the [Cheerio Crawler](../examples/cheerio-crawler) example.--## Request queue--The request queue is a storage of URLs to crawl. The queue is used for the deep crawling of websites, where you start with several URLs and then-recursively follow links to other pages. The data structure supports both breadth-first and depth-first crawling orders.--Each actor run is associated with a **default request queue**, which is created exclusively for the actor run. Typically, it is used to store URLs to-crawl in the specific actor run. Its usage is optional.--In Apify SDK, the request queue is represented by the [`RequestQueue`](../api/request-queue) class.--In local configuration, the request queue data is stored in the directory specified by the `APIFY_LOCAL_STORAGE_DIR` environment variable as follows:--```-{APIFY_LOCAL_STORAGE_DIR}/request_queues/{QUEUE_ID}/{STATE}/{NUMBER}.json-```--Note that `{QUEUE_ID}` is the name or ID of the request queue. The default queue has ID `default`, unless you override it by setting the-`APIFY_DEFAULT_REQUEST_QUEUE_ID` environment variable. Each request in the queue is stored as a separate JSON file, where `{STATE}` is either-`handled` or `pending`, and `{NUMBER}` is an integer indicating the position of the request in the queue.--The following code demonstrates basic operations of the request queue:--```javascript-// Open the default request queue associated with the actor run-const queue = await Apify.openRequestQueue();--// Open a named request queue-const queueWithName = await Apify.openRequestQueue('some-name');--// Enqueue few requests-await queue.addRequest({ url: 'http://example.com/aaa' });-await queue.addRequest({ url: 'http://example.com/bbb' });-await queue.addRequest({ url: 'http://example.com/foo/bar' }, { forefront: true });--// Get requests from queue-const request1 = await queue.fetchNextRequest();-const request2 = await queue.fetchNextRequest();-const request3 = await queue.fetchNextRequest();--// Mark a request as handled-await queue.markRequestHandled(request1);--// If processing fails then reclaim the request back to the queue, so that it's crawled again-await queue.reclaimRequest(request2);-```--To see how to use the request queue with a crawler, see the [Puppeteer Crawler](../examples/puppeteer-crawler) example.diff --git a/src/crawlers/basic_crawler.js b/src/crawlers/basic_crawler.jsindex c7508ba66..a2f512306 100644--- a/src/crawlers/basic_crawler.js+++ b/src/crawlers/basic_crawler.js@@ -118,7 +118,7 @@ const SAFE_MIGRATION_WAIT_MILLIS = 20000;* `BasicCrawler` is a low-level tool that requires the user to implement the page* download and data extraction functionality themselves.* If you want a crawler that already facilitates this functionality,- * please consider using {@link PuppeteerCrawler} or {@link CheerioCrawler}.+ * please consider using {@link CheerioCrawler}, {@link PuppeteerCrawler} or {@link PlaywrightCrawler}.** `BasicCrawler` invokes the user-provided {@link BasicCrawlerOptions.handleRequestFunction}* for each {@link Request} object, which represents a single URL to crawl.diff --git a/src/crawlers/cheerio_crawler.js b/src/crawlers/cheerio_crawler.jsindex acec9e794..8fb21c643 100644--- a/src/crawlers/cheerio_crawler.js+++ b/src/crawlers/cheerio_crawler.js@@ -251,7 +251,7 @@ const CHEERIO_OPTIMIZED_AUTOSCALED_POOL_OPTIONS = {** Since `CheerioCrawler` uses raw HTTP requests to download web pages,* it is very fast and efficient on data bandwidth. However, if the target website requires JavaScript- * to display the content, you might need to use {@link PuppeteerCrawler} instead,+ * to display the content, you might need to use {@link PuppeteerCrawler} or {@link PlaywrightCrawler} instead,* because it loads the pages using full-featured headless Chrome browser.** `CheerioCrawler` downloads each URL using a plain HTTP request,diff --git a/src/request.js b/src/request.jsindex f396f49f2..fc749147c 100644--- a/src/request.js+++ b/src/request.js@@ -39,7 +39,7 @@ const requestOptionalPredicates = {* Each `Request` instance has the `uniqueKey` property, which can be either specified* manually in the constructor or generated automatically from the URL. Two requests with the same `uniqueKey`* are considered as pointing to the same web resource. This behavior applies to all Apify SDK classes,- * such as {@link RequestList}, {@link RequestQueue} or {@link PuppeteerCrawler}.+ * such as {@link RequestList}, {@link RequestQueue}, {@link PuppeteerCrawler} or {@link PlaywrightCrawler}.** Example use:*@@ -67,7 +67,7 @@ const requestOptionalPredicates = {* An actually loaded URL after redirects, if present. HTTP redirects are guaranteed* to be included.*- * When using {@link PuppeteerCrawler}, meta tag and JavaScript redirects may,+ * When using {@link PuppeteerCrawler} or {@link PlaywrightCrawler}, meta tag and JavaScript redirects may,* or may not be included, depending on their nature. This generally means that redirects,* which happen immediately will most likely be included, but delayed redirects will not.* @property {string} uniqueKeydiff --git a/src/request_list.js b/src/request_list.jsindex 29676e12e..144bf54c6 100644--- a/src/request_list.js+++ b/src/request_list.js@@ -144,6 +144,8 @@ const CONTENT_TYPE_BINARY = 'application/octet-stream';/*** Represents a static list of URLs to crawl.* The URLs can be provided either in code or parsed from a text file hosted on the web.+ * `RequestList` is used by {@link BasicCrawler}, {@link CheerioCrawler}, {@link PuppeteerCrawler}+ * and {@link PlaywrightCrawler} as a source of URLs to crawl.** Each URL is represented using an instance of the {@link Request} class.* The list can only contain unique URLs. More precisely, it can only contain `Request` instances@@ -154,10 +156,12 @@ const CONTENT_TYPE_BINARY = 'application/octet-stream';** Once you create an instance of `RequestList`, you need to call the {@link RequestList#initialize} function* before the instance can be used. After that, no more URLs can be added to the list.- *- * `RequestList` is used by {@link BasicCrawler}, {@link CheerioCrawler}- * and {@link PuppeteerCrawler} as a source of URLs to crawl.* Unlike {@link RequestQueue}, `RequestList` is static but it can contain even millions of URLs.+ * > Note that `RequestList` can be used together with `RequestQueue` by the same crawler.+ * > In such cases, each request from `RequestList` is enqueued into `RequestQueue` first and then consumed from the latter.+ * > This is necessary to avoid the same URL being processed more than once (from the list first and then possibly from the queue).+ * > In practical terms, such a combination can be useful when there is a large number of initial URLs,+ * > but more URLs would be added dynamically by the crawler.** `RequestList` has an internal state where it stores information about which requests were already handled,* which are in progress and which were reclaimed. The state may be automatically persisted to the defaultdiff --git a/src/storages/request_queue.js b/src/storages/request_queue.jsindex 15c9d932f..cd60cfcab 100644--- a/src/storages/request_queue.js+++ b/src/storages/request_queue.js@@ -75,8 +75,8 @@ export const getRequestId = (uniqueKey) => {* Do not instantiate this class directly, use the* {@link Apify#openRequestQueue} function instead.*- * `RequestQueue` is used by {@link BasicCrawler}, {@link CheerioCrawler}- * and {@link PuppeteerCrawler} as a source of URLs to crawl.+ * `RequestQueue` is used by {@link BasicCrawler}, {@link CheerioCrawler}, {@link PuppeteerCrawler}+ * and {@link PlaywrightCrawler} as a source of URLs to crawl.* Unlike {@link RequestList}, `RequestQueue` supports dynamic adding and removing of requests.* On the other hand, the queue is not optimized for operations that add or remove a large number of URLs in a batch.*diff --git a/website/i18n/en.json b/website/i18n/en.jsonindex 839eb2743..80cb4c9a6 100644--- a/website/i18n/en.json+++ b/website/i18n/en.json@@ -143,9 +143,6 @@"guides/apify-platform": {"title": "Apify Platform"},- "guides/data-storage": {- "title": "Data Storage"- },"guides/docker-images": {"title": "Running in Docker"},@@ -164,6 +161,12 @@"guides/quick-start": {"title": "Quick Start"},+ "guides/request-storage": {+ "title": "Request Storage"+ },+ "guides/result-storage": {+ "title": "Result Storage"+ },"guides/session-management": {"title": "Session Management"},diff --git a/website/sidebars.json b/website/sidebars.jsonindex 884eb8ad3..b25751d14 100644--- a/website/sidebars.json+++ b/website/sidebars.json@@ -5,7 +5,8 @@"guides/quick-start","guides/apify-platform","guides/getting-started",- "guides/data-storage",+ "guides/request-storage",+ "guides/result-storage","guides/environment-variables","guides/proxy-management","guides/session-management",