Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

refactor!: Finalize transition to IRequestManager and IRequestLoader (#3719)

Jan Buchar404132221d59
diff --git a/docs/guides/request_loaders.mdx b/docs/guides/request_loaders.mdx
new file mode 100644
index 000000000..23b7444f7
--- /dev/null
+++ b/docs/guides/request_loaders.mdx
@@ -0,0 +1,179 @@
+---
+id: request-loaders
+title: Request loaders
+description: How to manage the requests your crawler will go through.
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+import RlBasicSource from '!!raw-loader!./request_loaders_rl_basic.ts';
+import SitemapBasicSource from '!!raw-loader!./request_loaders_sitemap_basic.ts';
+import RlTandemExplicitSource from '!!raw-loader!./request_loaders_rl_tandem_explicit.ts';
+import RlTandemHelperSource from '!!raw-loader!./request_loaders_rl_tandem_helper.ts';
+import SitemapTandemExplicitSource from '!!raw-loader!./request_loaders_sitemap_tandem_explicit.ts';
+import SitemapTandemHelperSource from '!!raw-loader!./request_loaders_sitemap_tandem_helper.ts';
+
+Request loaders extend the functionality of the <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>, providing additional tools for managing URLs and requests. If you are new to Crawlee and unfamiliar with the <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>, consider starting with the [Request storage](./request-storage) guide first. Request loaders define how requests are fetched and stored, enabling various use cases such as reading URLs from a static list, a sitemap, an external API, or combining multiple sources together.
+
+## Overview
+
+The request loader abstractions are built around two interfaces and a couple of helpers:
+
+- <ApiLink to="core/interface/IRequestLoader">`IRequestLoader`</ApiLink>: The base interface for reading requests in a crawl.
+- <ApiLink to="core/interface/IRequestManager">`IRequestManager`</ApiLink>: Extends `IRequestLoader` with write capabilities (adding and reclaiming requests).
+- <ApiLink to="core/class/RequestManagerTandem">`RequestManagerTandem`</ApiLink>: Combines a read-only `IRequestLoader` with a writable `IRequestManager`.
+
+And the concrete request loader implementations:
+
+- <ApiLink to="core/class/RequestList">`RequestList`</ApiLink>: A lightweight implementation for managing a static list of URLs.
+- <ApiLink to="core/class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink>: A specialized loader that reads URLs from XML and plain-text sitemaps following the [Sitemaps protocol](https://www.sitemaps.org/protocol.html), with filtering capabilities.
+
+Below is a class diagram that illustrates the relationships between these components and the <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>:
+
+```mermaid
+---
+config:
+ class:
+ hideEmptyMembersBox: true
+---
+
+classDiagram
+
+%% ========================
+%% Abstract interfaces
+%% ========================
+
+class IRequestLoader {
+ <<interface>>
+ + getTotalCount()
+ + getPendingCount()
+ + getHandledCount()
+ + fetchNextRequest()
+ + markRequestHandled()
+ + isEmpty()
+ + isFinished()
+ + toTandem()
+}
+
+class IRequestManager {
+ <<interface>>
+ + addRequest()
+ + addRequestsBatched()
+ + reclaimRequest()
+ + purge()
+}
+
+%% ========================
+%% Concrete classes
+%% ========================
+
+class RequestQueue
+
+class RequestList
+
+class SitemapRequestLoader
+
+class RequestManagerTandem
+
+%% ========================
+%% Inheritance arrows
+%% ========================
+
+IRequestLoader <|-- IRequestManager
+IRequestLoader <|.. RequestList
+IRequestLoader <|.. SitemapRequestLoader
+IRequestManager <|.. RequestQueue
+IRequestManager <|.. RequestManagerTandem
+```
+
+:::info Crawler usage
+
+A crawler reads its requests from a single <ApiLink to="core/interface/IRequestManager">`IRequestManager`</ApiLink>, passed via the `requestManager` option. A <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink> is itself a request manager, so it can be passed directly. A read-only loader (such as <ApiLink to="core/class/RequestList">`RequestList`</ApiLink>) cannot — combine it with a queue into a tandem first, see the [Request manager tandem](#request-manager-tandem) section below.
+
+:::
+
+## Request loaders
+
+The <ApiLink to="core/interface/IRequestLoader">`IRequestLoader`</ApiLink> interface defines the foundation for fetching requests during a crawl. It provides methods for basic operations like retrieving the next request, marking requests as handled, and checking whether the loader is empty or finished. It is intentionally **read-only** — it does not allow adding new requests. Concrete implementations such as <ApiLink to="core/class/RequestList">`RequestList`</ApiLink> build on this interface to handle specific scenarios. You can create your own custom loader that reads from an external file, web endpoint, database, or any other data source.
+
+### Request list
+
+The <ApiLink to="core/class/RequestList">`RequestList`</ApiLink> manages a static list of URLs to crawl. The list is created for a single crawler run and, unlike a queue, cannot have requests added to or removed from it after initialization. It can hold a large number of URLs (even millions) with significantly lower overhead than enqueueing them one by one.
+
+Here is a basic example of working with the <ApiLink to="core/class/RequestList">`RequestList`</ApiLink>:
+
+<CodeBlock language="typescript">
+ {RlBasicSource}
+</CodeBlock>
+
+### Sitemap request loader
+
+The <ApiLink to="core/class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> is a specialized request loader that reads URLs from sitemaps following the [Sitemaps protocol](https://www.sitemaps.org/protocol.html). It supports both XML and plain-text sitemap formats and is particularly useful when you want to crawl a website systematically by following its sitemap structure. Loading happens in the background, so crawling can start before the sitemap is fully parsed.
+
+:::note
+
+The `SitemapRequestLoader` is designed specifically for sitemaps that follow the standard Sitemaps protocol. HTML pages containing links are not supported by this loader — those should be handled by regular crawlers using the `enqueueLinks` functionality.
+
+:::
+
+The loader supports filtering URLs using glob patterns and regular expressions, allowing you to include or exclude specific types of URLs.
+
+<CodeBlock language="typescript">
+ {SitemapBasicSource}
+</CodeBlock>
+
+## Request managers
+
+The <ApiLink to="core/interface/IRequestManager">`IRequestManager`</ApiLink> interface extends `IRequestLoader` with **write** capabilities. In addition to reading requests, a request manager can add new requests and reclaim failed ones. This is essential for dynamic crawling, where new URLs emerge during the crawl, or when requests fail and need to be retried. The <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink> is the primary built-in request manager — see the [Request storage](./request-storage) guide for details.
+
+## Request manager tandem
+
+The <ApiLink to="core/class/RequestManagerTandem">`RequestManagerTandem`</ApiLink> class combines the read-only capabilities of an `IRequestLoader` (like <ApiLink to="core/class/RequestList">`RequestList`</ApiLink>) with the read-write capabilities of an `IRequestManager` (like <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>). This is useful when you need to load initial requests from a static source (such as a file, sitemap, or database) and also dynamically add or retry requests during the crawl.
+
+Under the hood, the tandem checks whether the read-only loader still has pending requests. If so, each request from the loader is transferred to the manager (the queue) before being processed. Any newly added or reclaimed requests go directly to the manager side. Because every request passes through the queue, deduplication and retries are handled consistently and a single URL is not crawled multiple times.
+
+The easiest way to build a tandem is the <ApiLink to="core/interface/IRequestLoader#toTandem">`toTandem()`</ApiLink> helper available on the loaders. Called without arguments, it pairs the loader with the default <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>; you can also pass a specific request manager to use instead.
+
+### Request list with request queue
+
+This setup is useful when you have a static list of URLs to crawl, but also need to handle dynamic requests discovered during the crawl. Requests from the <ApiLink to="core/class/RequestList">`RequestList`</ApiLink> are processed first by being enqueued into the <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>, which handles persistence and retries.
+
+<Tabs groupId="request_manager_tandem">
+ <TabItem value="request_manager_tandem_helper" label="Using toTandem helper" default>
+ <CodeBlock language="typescript">
+ {RlTandemHelperSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="request_manager_tandem_explicit" label="Explicit usage">
+ <CodeBlock language="typescript">
+ {RlTandemExplicitSource}
+ </CodeBlock>
+ </TabItem>
+</Tabs>
+
+### Sitemap request loader with request queue
+
+Similarly, you can combine a <ApiLink to="core/class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> with a <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>. This is particularly useful when you want to crawl URLs from a sitemap while also handling dynamic requests discovered during the crawl. URLs from the sitemap are processed first by being enqueued into the queue, which handles persistence and retries.
+
+<Tabs groupId="sitemap_request_manager_tandem">
+ <TabItem value="sitemap_request_manager_tandem_helper" label="Using toTandem helper" default>
+ <CodeBlock language="typescript">
+ {SitemapTandemHelperSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="sitemap_request_manager_tandem_explicit" label="Explicit usage">
+ <CodeBlock language="typescript">
+ {SitemapTandemExplicitSource}
+ </CodeBlock>
+ </TabItem>
+</Tabs>
+
+## Conclusion
+
+This guide introduced the request loader abstractions: the read-only <ApiLink to="core/interface/IRequestLoader">`IRequestLoader`</ApiLink>, the writable <ApiLink to="core/interface/IRequestManager">`IRequestManager`</ApiLink>, and the <ApiLink to="core/class/RequestManagerTandem">`RequestManagerTandem`</ApiLink> that combines them, along with the <ApiLink to="core/class/RequestList">`RequestList`</ApiLink> and <ApiLink to="core/class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> implementations. You also saw how to pair a loader with a queue using the `toTandem()` helper to handle both static and dynamically discovered requests.
+
+If you have questions or need assistance, feel free to reach out on our [GitHub](https://github.com/apify/crawlee) or join our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping!
diff --git a/docs/guides/request_loaders_rl_basic.ts b/docs/guides/request_loaders_rl_basic.ts
new file mode 100644
index 000000000..c974b668b
--- /dev/null
+++ b/docs/guides/request_loaders_rl_basic.ts
@@ -0,0 +1,15 @@
+import { RequestList } from 'crawlee';
+
+// Open a request list with a static set of URLs.
+// The name is used to persist the list's state in the default key-value store.
+const requestList = await RequestList.open('my-list', [
+ 'https://crawlee.dev/',
+ 'https://crawlee.dev/docs',
+ 'https://crawlee.dev/api',
+]);
+
+// Iterate over the requests manually (a crawler does this for you under the hood).
+for await (const request of requestList) {
+ console.log(request.url);
+ await requestList.markRequestHandled(request);
+}
diff --git a/docs/guides/request_loaders_rl_tandem_explicit.ts b/docs/guides/request_loaders_rl_tandem_explicit.ts
new file mode 100644
index 000000000..8014ddc79
--- /dev/null
+++ b/docs/guides/request_loaders_rl_tandem_explicit.ts
@@ -0,0 +1,21 @@
+import { CheerioCrawler, RequestList, RequestManagerTandem, RequestQueue } from 'crawlee';
+
+// A static list of URLs to start from (can hold millions of URLs).
+const requestList = await RequestList.open('my-list', ['https://crawlee.dev/', 'https://crawlee.dev/docs']);
+
+// A writable queue that holds requests discovered during the crawl.
+const requestQueue = await RequestQueue.open();
+
+// Combine them: the tandem reads from the list first, transferring each request
+// into the queue, and lets you enqueue new requests during the crawl.
+const requestManager = new RequestManagerTandem(requestList, requestQueue);
+
+const crawler = new CheerioCrawler({
+ requestManager,
+ async requestHandler({ enqueueLinks }) {
+ // Newly discovered links go to the queue side of the tandem.
+ await enqueueLinks();
+ },
+});
+
+await crawler.run();
diff --git a/docs/guides/request_loaders_rl_tandem_helper.ts b/docs/guides/request_loaders_rl_tandem_helper.ts
new file mode 100644
index 000000000..8637cb510
--- /dev/null
+++ b/docs/guides/request_loaders_rl_tandem_helper.ts
@@ -0,0 +1,17 @@
+import { CheerioCrawler, RequestList } from 'crawlee';
+
+// A static list of URLs to start from.
+const requestList = await RequestList.open('my-list', ['https://crawlee.dev/', 'https://crawlee.dev/docs']);
+
+// `toTandem()` is a shortcut that pairs the loader with a request queue.
+// Without arguments it opens the default `RequestQueue`.
+const requestManager = await requestList.toTandem();
+
+const crawler = new CheerioCrawler({
+ requestManager,
+ async requestHandler({ enqueueLinks }) {
+ await enqueueLinks();
+ },
+});
+
+await crawler.run();
diff --git a/docs/guides/request_loaders_sitemap_basic.ts b/docs/guides/request_loaders_sitemap_basic.ts
new file mode 100644
index 000000000..fff7c2867
--- /dev/null
+++ b/docs/guides/request_loaders_sitemap_basic.ts
@@ -0,0 +1,14 @@
+import { SitemapRequestLoader } from 'crawlee';
+
+// Open a sitemap request list. The sitemap is fetched and parsed in the background,
+// so crawling can start before the whole sitemap is loaded.
+const sitemapRequestLoader = await SitemapRequestLoader.open({
+ sitemapUrls: ['https://crawlee.dev/sitemap.xml'],
+ // Optionally filter the URLs read from the sitemap:
+ // globs: ['https://crawlee.dev/docs/**'],
+});
+
+for await (const request of sitemapRequestLoader) {
+ console.log(request.url);
+ await sitemapRequestLoader.markRequestHandled(request);
+}
diff --git a/docs/guides/request_loaders_sitemap_tandem_explicit.ts b/docs/guides/request_loaders_sitemap_tandem_explicit.ts
new file mode 100644
index 000000000..48d2f936e
--- /dev/null
+++ b/docs/guides/request_loaders_sitemap_tandem_explicit.ts
@@ -0,0 +1,20 @@
+import { CheerioCrawler, RequestManagerTandem, RequestQueue, SitemapRequestLoader } from 'crawlee';
+
+// Read the initial URLs from a sitemap.
+const sitemapRequestLoader = await SitemapRequestLoader.open({
+ sitemapUrls: ['https://crawlee.dev/sitemap.xml'],
+});
+
+// A writable queue for requests discovered during the crawl.
+const requestQueue = await RequestQueue.open();
+
+const requestManager = new RequestManagerTandem(sitemapRequestLoader, requestQueue);
+
+const crawler = new CheerioCrawler({
+ requestManager,
+ async requestHandler({ enqueueLinks }) {
+ await enqueueLinks();
+ },
+});
+
+await crawler.run();
diff --git a/docs/guides/request_loaders_sitemap_tandem_helper.ts b/docs/guides/request_loaders_sitemap_tandem_helper.ts
new file mode 100644
index 000000000..bcf1c2ea0
--- /dev/null
+++ b/docs/guides/request_loaders_sitemap_tandem_helper.ts
@@ -0,0 +1,18 @@
+import { CheerioCrawler, SitemapRequestLoader } from 'crawlee';
+
+// Read the initial URLs from a sitemap.
+const sitemapRequestLoader = await SitemapRequestLoader.open({
+ sitemapUrls: ['https://crawlee.dev/sitemap.xml'],
+});
+
+// Pair the loader with the default `RequestQueue` via the `toTandem()` shortcut.
+const requestManager = await sitemapRequestLoader.toTandem();
+
+const crawler = new CheerioCrawler({
+ requestManager,
+ async requestHandler({ enqueueLinks }) {
+ await enqueueLinks();
+ },
+});
+
+await crawler.run();
diff --git a/docs/guides/request_storage.mdx b/docs/guides/request_storage.mdx
index 8da5489b5..11aed2831 100644
--- a/docs/guides/request_storage.mdx
+++ b/docs/guides/request_storage.mdx
@@ -14,7 +14,6 @@ import BasicOperationsSource from '!!raw-loader!./request_storage_queue_basic.ts
import CrawlerExplicitSource from '!!raw-loader!./request_storage_queue_crawler_explicit.ts';
import CrawlerSource from '!!raw-loader!./request_storage_queue_crawler.ts';
-import RequestQueueListSource from '!!raw-loader!./request_storage_queue_list.ts';
import RequestQueueAddRequestsSource from '!!raw-loader!./request_storage_queue_only.ts';
Crawlee has several request storage types that are useful for specific tasks. The requests are stored on local disk to a directory defined by the `CRAWLEE_STORAGE_DIR` environment variable. If this variable is not defined, by default Crawlee sets `CRAWLEE_STORAGE_DIR` to `./storage` in the current working directory.
@@ -67,71 +66,17 @@ The following code demonstrates the usage of the request queue:
To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/js/docs/examples/puppeteer-crawler) example.
-## Request list
+The request queue is not optimized for adding numerous URLs in a single batch — historically, requests were added one by one. To enqueue a large set of initial URLs efficiently, use the <ApiLink to="core/class/RequestQueue#addRequests">`addRequests()`</ApiLink> method (or simply pass the URLs to `crawler.run()`), which adds requests in batches:
-The request list is not a storage per se - it represents the list of URLs to crawl that is stored in a crawler run memory (or optionally in default [Key-Value Store](../guides/result-storage#key-value-store) associated with the run, if specified). The list is used for the crawling of a large number of URLs, when we know all the URLs which should be visited by the crawler and no URLs would be added during the run. The URLs can be provided either in code or parsed from a text file hosted on the web.
+<CodeBlock language="js">
+ {RequestQueueAddRequestsSource}
+</CodeBlock>
-Request list is created exclusively for the crawler run and only if its usage is explicitly specified in the code. Its usage is optional.
+## Reading requests from other sources
-In Crawlee, the request list is represented by the <ApiLink to="core/class/RequestList">`RequestList`</ApiLink> class.
+Sometimes you don't want to start from a dynamic queue, but from a static list of URLs (for example, parsed from a file) or from a website's sitemap. Crawlee provides **request loaders** for these read-only sources — <ApiLink to="core/class/RequestList">`RequestList`</ApiLink> and <ApiLink to="core/class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> — which can be combined with a request queue when you also need to enqueue requests discovered during the crawl.
-The following code demonstrates basic operations of the request list:
-
-```javascript
-import { RequestList, PuppeteerCrawler } from 'crawlee';
-
-// Prepare the sources array with URLs to visit
-const sources = [
- { url: 'http://www.example.com/page-1' },
- { url: 'http://www.example.com/page-2' },
- { url: 'http://www.example.com/page-3' },
-];
-
-// Open the request list.
-// List name is used to persist the sources and the list state in the key-value store
-const requestList = await RequestList.open('my-list', sources);
-
-// The crawler will automatically process requests from the list
-// It's used the same way for Cheerio /Playwright crawlers.
-const crawler = new PuppeteerCrawler({
- requestList,
- async requestHandler({ page, request }) {
- // Process the page (extract data, take page screenshot, etc).
- // No more requests could be added to the request list here
- },
-});
-```
-
-## Which one to choose?
-
-When using Request queue - we would normally have several start URLs (e.g. category pages on e-commerce website) and then recursively add more (e.g. individual item pages) programmatically to the queue, it supports dynamic adding and removing of requests. No more URLs can be added to Request list after its initialization as it is immutable, URLs cannot be removed from the list either.
-
-On the other hand, the Request queue is not optimized for adding or removing numerous URLs in a batch. This is technically possible, but requests are added one by one to the queue, and thus it would take significant time with a larger number of requests. Request list however can contain even millions of URLs, and it would take significantly less time to add them to the list, compared to the queue.
-
-Note that Request queue and Request list can be used together by the same crawler. In such cases, each request from the Request list is enqueued into the Request queue first (to the foremost position in the queue, even if Request queue is not empty) and then consumed from the latter. This is necessary to avoid the same URL being processed more than once (from the list first and then possibly from the queue). In practical terms, such a combination can be useful when there are numerous initial URLs, but more URLs would be added dynamically by the crawler.
-
-:::tip
-
-In Crawlee, there is not much need to combine the request queue together with the request list (although it's technically possible).
-
-Previously there was no way to add the initial requests to the queue in batches (to add an array of requests), i.e. we could have only added the requests one by one to the queue with the help of <ApiLink to="core/class/RequestQueue#addRequest">`addRequest()`</ApiLink> function.
-
-However, now we could use the <ApiLink to="core/class/RequestQueue#addRequests">`addRequests()`</ApiLink> function, which adds requests in batches. Thus, instead of combining the request queue and the request list, we can use only the request queue for such use-cases now. See the examples below.
-
-:::
-
-<Tabs groupId="queue_list">
- <TabItem value="add_requests" label="Request Queue" default>
- <CodeBlock language="js">
- {RequestQueueAddRequestsSource}
- </CodeBlock>
- </TabItem>
- <TabItem value="queue_list" label="Request Queue + Request List">
- <CodeBlock language="js">
- {RequestQueueListSource}
- </CodeBlock>
- </TabItem>
-</Tabs>
+See the dedicated [Request loaders](./request-loaders) guide for details on loaders, request managers, and how to combine them with a queue into a <ApiLink to="core/class/RequestManagerTandem">`RequestManagerTandem`</ApiLink>.
## Cleaning up the storages
diff --git a/docs/guides/request_storage_queue_list.ts b/docs/guides/request_storage_queue_list.ts
deleted file mode 100644
index 37de17eec..000000000
--- a/docs/guides/request_storage_queue_list.ts
+++ /dev/null
@@ -1,42 +0,0 @@
-// This is technically correct, but
-// we need to explicitly open/use both the request queue and the request list.
-// We suggest using the request queue and batch add the requests instead.
-
-import { RequestList, RequestQueue, PuppeteerCrawler } from 'crawlee';
-
-// Prepare the sources array with URLs to visit (it can contain millions of URLs)
-const sources = [
- { url: 'http://www.example.com/page-1' },
- { url: 'http://www.example.com/page-2' },
- { url: 'http://www.example.com/page-3' },
- // ...
-];
-
-// Open the request list with the initial sources array
-const requestList = await RequestList.open('my-list', sources);
-
-// Open the default request queue. It's not necessary to add any requests to the queue
-const requestQueue = await RequestQueue.open();
-
-// The crawler will automatically process requests from the list and the queue.
-// It's used the same way for Cheerio/Playwright crawlers
-const crawler = new PuppeteerCrawler({
- requestList,
- requestQueue,
- // Each request from the request list is enqueued to the request queue one by one.
- // At this point request with the same URL would exist in the list and the queue
- async requestHandler({ enqueueLinks }) {
- // Add new request to the queue
- await crawler.addRequests(['http://www.example.com/new-page']);
-
- // Add links found on page to the queue
- await enqueueLinks();
-
- // The requests above would be added to the queue (but not to the list)
- // and would be processed after the request list is empty.
- // No more requests could be added to the list here
- },
-});
-
-// Run the crawler
-await crawler.run();
diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.md
index ac2b19c05..b9f1e61b3 100644
--- a/docs/upgrading/upgrading_v4.md
+++ b/docs/upgrading/upgrading_v4.md
@@ -616,6 +616,127 @@ const dataset = await Dataset.open();
The same change applies to `CrawlingContext.getKeyValueStore()` and `CrawlingContext.pushData()` — both now accept `string | StorageIdentifier` for identifying the target storage.
+## Request loaders and managers
+
+The request loader/manager interfaces have been reworked to mirror the abstractions in Crawlee for Python. See the new [Request loaders](../guides/request-loaders) guide for the full picture.
+
+### `IRequestList` renamed to `IRequestLoader`
+
+The `IRequestList` interface has been renamed to `IRequestLoader` and is now the read-only base interface implemented by `RequestList` and `SitemapRequestLoader`. The writable `IRequestManager` interface now **extends** `IRequestLoader` with the request-adding and reclaiming surface (`addRequest`, `addRequestsBatched`, `reclaimRequest`, optional `purge`). There is no `IRequestList` alias — update your imports and type references to `IRequestLoader` (or `IRequestManager` if you need the write surface).
+
+### Loader interface surface changes
+
+The harmonized loader interface differs from the old `IRequestList` in a few ways:
+
+| Before (v3) | After (v4) |
+|---|---|
+| `length(): number` | `getTotalCount(): Promise<number>` (renamed and now async) |
+| _(n/a)_ | `getPendingCount(): Promise<number>` (new) |
+| `handledCount(): number` | `getHandledCount(): Promise<number>` (renamed and now async) |
+| `reclaimRequest()` on the interface | Removed from the read-only loaders entirely; reclaiming is a write operation that lives only on `IRequestManager` (e.g. `RequestQueue`, `RequestManagerTandem`) |
+| `inProgress: Set<string>` on the interface | Removed from the interface |
+| `persistState(): Promise<void>` (required) | `persistState?(): Promise<void>` (optional) |
+| _(n/a)_ | `toTandem?(requestManager?)` (new) |
+
+`RequestList.length()` and `RequestList.handledCount()` (and their `SitemapRequestLoader` counterparts) were renamed to `getTotalCount()` and `getHandledCount()` and are now `async` — `await` them.
+
+**Before:**
+```typescript
+const total = requestList.length();
+const handled = requestList.handledCount();
+```
+
+**After:**
+```typescript
+const total = await requestList.getTotalCount();
+const handled = await requestList.getHandledCount();
+```
+
+### Combining a list and a queue: `toTandem()`
+
+`RequestList` and `SitemapRequestLoader` now expose a `toTandem()` helper that pairs the read-only loader with a writable request manager (the default `RequestQueue` if none is passed), producing a `RequestManagerTandem` you can hand to a crawler via the new `requestManager` option:
+
+```typescript
+import { CheerioCrawler, RequestList } from 'crawlee';
+
+const requestList = await RequestList.open('my-list', ['https://example.com']);
+
+const crawler = new CheerioCrawler({
+ requestManager: await requestList.toTandem(),
+ requestHandler: async ({ enqueueLinks }) => {
+ await enqueueLinks();
+ },
+});
+```
+
+### `SitemapRequestList` renamed to `SitemapRequestLoader`
+
+The `SitemapRequestList` class (and its `SitemapRequestListOptions` type) have been renamed to `SitemapRequestLoader` and `SitemapRequestLoaderOptions` to match the loader terminology. Update your imports and type references accordingly:
+
+```typescript
+// Before
+import { SitemapRequestList } from 'crawlee';
+const loader = await SitemapRequestList.open({ sitemapUrls: ['https://example.com/sitemap.xml'] });
+
+// After
+import { SitemapRequestLoader } from 'crawlee';
+const loader = await SitemapRequestLoader.open({ sitemapUrls: ['https://example.com/sitemap.xml'] });
+```
+
+The default `KeyValueStore` key used to persist the loader's state was also renamed from `SITEMAP_REQUEST_LIST_STATE` to `SITEMAP_REQUEST_LOADER_STATE`. State persisted under the old key by a v3 run will **not** be picked up after upgrading, so any in-flight sitemap crawl that migrates across the upgrade will restart from the beginning. If you need to preserve state, either finish the crawl before upgrading or pass an explicit `persistStateKey`.
+
+### Crawler `requestList` / `requestQueue` options deprecated in favor of `requestManager`
+
+The crawler now reads its requests from a single `requestManager` (any `IRequestManager`, including a `RequestQueue`). The `requestList` and `requestQueue` constructor options are **deprecated** but still accepted as sugar:
+
+- `requestQueue` alone → used directly as the manager.
+- `requestList` + `requestQueue` → combined into a `RequestManagerTandem` automatically.
+- `requestList` alone → combined with a lazily-opened default queue into a tandem.
+
+```typescript
+// Before
+const crawler = new CheerioCrawler({ requestList, requestQueue });
+
+// After
+const crawler = new CheerioCrawler({ requestManager: new RequestManagerTandem(requestList, requestQueue) });
+// or, equivalently
+const crawler = new CheerioCrawler({ requestManager: await requestList.toTandem(requestQueue) });
+```
+
+A lone `requestList` now runs through a tandem over an auto-opened queue (rather than a read-only adapter). This means retries and `maxRequestsPerCrawl` accounting for that path now follow queue semantics.
+
+### `BasicCrawler.requestList` and `BasicCrawler.requestQueue` fields removed
+
+The public `requestList` and `requestQueue` instance fields are gone. The crawler exposes a single `protected requestManager?: IRequestManager` instead. Access the active manager via the new async `getRequestManager()` method.
+
+### `getRequestQueue()` deprecated in favor of `getRequestManager()`
+
+`BasicCrawler.getRequestQueue()` is deprecated. It still works as an alias, but now returns an `IRequestManager` that is no longer guaranteed to be a `RequestProvider`/`RequestQueue` (it may be a `RequestManagerTandem`). Use `getRequestManager()` instead.
+
+**Before:**
+```typescript
+const queue = await crawler.getRequestQueue();
+```
+
+**After:**
+```typescript
+const manager = await crawler.getRequestManager();
+```
+
+### `enqueueLinks` `requestQueue` option renamed to `requestManager`
+
+The standalone `enqueueLinks()` function and the click-elements enqueue helpers (`enqueueLinksByClickingElements` in `@crawlee/puppeteer` and `@crawlee/playwright`) now take a `requestManager` option instead of `requestQueue`:
+
+**Before:**
+```typescript
+await enqueueLinks({ urls, requestQueue });
+```
+
+**After:**
+```typescript
+await enqueueLinks({ urls, requestManager });
+```
+
## `transformRequestFunction` precedence in `enqueueLinks`
The `transformRequestFunction` callback in `enqueueLinks` now runs **after** URL pattern filtering (`globs`, `regexps`, `pseudoUrls`) instead of before. This means it has the highest priority and can overwrite any request options set by patterns or the global `label` option.
diff --git a/packages/basic-crawler/src/internals/basic-crawler.ts b/packages/basic-crawler/src/internals/basic-crawler.ts
index 510d521df..ef5a90541 100644
--- a/packages/basic-crawler/src/internals/basic-crawler.ts
+++ b/packages/basic-crawler/src/internals/basic-crawler.ts
@@ -13,7 +13,7 @@ import type {
EventManager,
FinalStatistics,
GetUserDataFromRequest,
- IRequestList,
+ IRequestLoader,
IRequestManager,
ProxyConfiguration,
Request,
@@ -48,7 +48,6 @@ import {
NonRetryableError,
purgeDefaultStorages,
RequestHandlerError,
- RequestListAdapter,
RequestManagerTandem,
RequestProvider,
RequestQueue,
@@ -191,25 +190,27 @@ export interface BasicCrawlerOptions<
/**
* Static list of URLs to be processed.
- * If not provided, the crawler will open the default request queue when the {@apilink BasicCrawler.addRequests|`crawler.addRequests()`} function is called.
- * > Alternatively, `requests` parameter of {@apilink BasicCrawler.run|`crawler.run()`} could be used to enqueue the initial requests -
- * it is a shortcut for running `crawler.addRequests()` before the `crawler.run()`.
+ *
+ * @deprecated Use the `requestManager` option instead. To combine a read-only loader (such as a `RequestList`)
+ * with a writable queue, build a tandem with {@apilink IRequestLoader.toTandem|`requestList.toTandem(requestQueue)`}
+ * and pass the result as `requestManager`. When both `requestList` and `requestQueue` are provided, they are
+ * combined into a tandem automatically.
*/
- requestList?: IRequestList;
+ requestList?: IRequestLoader;
/**
* Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
- * If not provided, the crawler will open the default request queue when the {@apilink BasicCrawler.addRequests|`crawler.addRequests()`} function is called.
- * > Alternatively, `requests` parameter of {@apilink BasicCrawler.run|`crawler.run()`} could be used to enqueue the initial requests -
- * it is a shortcut for running `crawler.addRequests()` before the `crawler.run()`.
+ *
+ * @deprecated Use the `requestManager` option instead. A `RequestQueue` is itself a request manager, so you can
+ * pass it directly as `requestManager`.
*/
requestQueue?: RequestProvider;
/**
- * Allows explicitly configuring a request manager. Mutually exclusive with the `requestQueue` and `requestList` options.
+ * Manager of requests that should be processed by the crawler. Mutually exclusive with the deprecated
+ * `requestQueue` and `requestList` options.
*
- * This enables explicitly configuring the crawler to use `RequestManagerTandem`, for instance.
- * If using this, the type of `BasicCrawler.requestQueue` may not be fully compatible with the `RequestProvider` class.
+ * If not provided, the crawler will open the default {@apilink RequestQueue} when it is first needed.
*/
requestManager?: IRequestManager;
@@ -477,15 +478,22 @@ export interface CrawlerExperiments {
*
* `BasicCrawler` invokes the user-provided {@apilink BasicCrawlerOptions.requestHandler|`requestHandler`}
* for each {@apilink Request} object, which represents a single URL to crawl.
- * The {@apilink Request} objects are fed from the {@apilink RequestList} or {@apilink RequestQueue}
- * instances provided by the {@apilink BasicCrawlerOptions.requestList|`requestList`} or {@apilink BasicCrawlerOptions.requestQueue|`requestQueue`}
- * constructor options, respectively. If neither `requestList` nor `requestQueue` options are provided,
- * the crawler will open the default request queue either when the {@apilink BasicCrawler.addRequests|`crawler.addRequests()`} function is called,
- * or if `requests` parameter (representing the initial requests) of the {@apilink BasicCrawler.run|`crawler.run()`} function is provided.
+ * The {@apilink Request} objects are fed from the {@apilink IRequestManager|request manager} provided via the
+ * {@apilink BasicCrawlerOptions.requestManager|`requestManager`} constructor option (a {@apilink RequestQueue} is
+ * itself a request manager). If no `requestManager` is provided, the crawler opens the default {@apilink RequestQueue}
+ * either when the {@apilink BasicCrawler.addRequests|`crawler.addRequests()`} function is called, or if the `requests`
+ * parameter (representing the initial requests) of the {@apilink BasicCrawler.run|`crawler.run()`} function is provided.
+ *
+ * To read requests from a read-only source such as a {@apilink RequestList} or {@apilink SitemapRequestLoader} while
+ * still being able to enqueue new ones, combine the loader with a queue into a {@apilink RequestManagerTandem} using
+ * {@apilink IRequestLoader.toTandem|`requestLoader.toTandem()`} and pass the result as `requestManager`. The tandem
+ * first processes URLs from the loader and automatically enqueues them into the queue, ensuring a single URL is not
+ * crawled multiple times.
*
- * If both {@apilink BasicCrawlerOptions.requestList|`requestList`} and {@apilink BasicCrawlerOptions.requestQueue|`requestQueue`} options are used,
- * the instance first processes URLs from the {@apilink RequestList} and automatically enqueues all of them
- * to the {@apilink RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
+ * > The legacy {@apilink BasicCrawlerOptions.requestList|`requestList`} and
+ * > {@apilink BasicCrawlerOptions.requestQueue|`requestQueue`} options are deprecated. They are still accepted and
+ * > folded into a single `requestManager` (combined into a tandem when both are given), but new code should use
+ * > `requestManager` directly.
*
* The crawler finishes if there are no more {@apilink Request} objects to crawl.
*
@@ -555,20 +563,9 @@ export class BasicCrawler<
readonly stats: Statistics;
/**
- * A reference to the underlying {@apilink RequestList} class that manages the crawler's {@apilink Request|requests}.
- * Only available if used by the crawler.
- */
- requestList?: IRequestList;
-
- /**
- * Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
- * A reference to the underlying {@apilink RequestQueue} class that manages the crawler's {@apilink Request|requests}.
- * Only available if used by the crawler.
- */
- requestQueue?: RequestProvider;
-
- /**
- * The main request-handling component of the crawler. It's initialized during the crawler startup.
+ * The main request-handling component of the crawler. It manages the requests that the crawler processes,
+ * combining any provided request loader and/or queue. It's initialized during the crawler startup or lazily
+ * via {@apilink BasicCrawler.getRequestManager|`getRequestManager()`}.
*/
protected requestManager?: IRequestManager;
@@ -587,11 +584,11 @@ export class BasicCrawler<
private ownedSessionPool?: SessionPool;
/**
- * Set when the crawler constructed its own {@apilink RequestQueue} (no `requestQueue`/`requestManager`
- * option was provided). The owned queue is purged (not dropped) between repeated `run()` calls.
- * A user-supplied queue is never purged by the crawler.
+ * Set when the crawler constructed its own request manager (no `requestManager`, `requestQueue`, or `requestList`
+ * option was provided). The owned manager is purged (not dropped) between repeated `run()` calls.
+ * A user-supplied manager is never purged by the crawler.
*/
- private ownedRequestQueue?: RequestProvider;
+ private ownedRequestManager?: IRequestManager;
/**
* A reference to the underlying {@apilink AutoscaledPool} class that manages the concurrency of the crawler.
@@ -760,7 +757,9 @@ export class BasicCrawler<
ow(options, 'BasicCrawlerOptions', ow.object.exactShape(BasicCrawler.optionsShape));
const {
+ // oxlint-disable-next-line typescript/no-deprecated -- still accepted and folded into `requestManager` for back-compat
requestList,
+ // oxlint-disable-next-line typescript/no-deprecated -- still accepted and folded into `requestManager` for back-compat
requestQueue,
requestManager,
maxRequestRetries = 3,
@@ -844,10 +843,17 @@ export class BasicCrawler<
);
}
this.requestManager = requestManager;
- this.requestQueue = requestManager as RequestProvider; // TODO(v4) - the cast is not fully legitimate here, but it's fine for internal usage by the BasicCrawler
- } else {
- this.requestList = requestList;
- this.requestQueue = requestQueue;
+ } else if (requestList !== undefined && requestQueue !== undefined) {
+ // Combine the read-only list with the writable queue into a tandem.
+ this.requestManager = new RequestManagerTandem(requestList, requestQueue);
+ } else if (requestQueue !== undefined) {
+ // A RequestQueue is itself a request manager.
+ this.requestManager = requestQueue;
+ } else if (requestList !== undefined) {
+ // A lone read-only `requestList` (deprecated option) is combined with a lazily-opened default queue
+ // into a tandem, so that its requests are read first and new ones can still be enqueued during the
+ // crawl. The queue is opened on first use; the tandem also forwards `persistState()` to the loader.
+ this.requestManager = new RequestManagerTandem(requestList, () => this.openOwnedRequestQueue());
}
this.httpClient = httpClient ?? new GotScrapingHttpClient({ logger: this.log });
@@ -882,11 +888,8 @@ export class BasicCrawler<
tryEnv(process.env.CRAWLEE_INTERNAL_TIMEOUT) ?? Math.max(this.requestHandlerTimeoutMillis * 2, 300e3);
// override the default internal timeout of request queue to respect `requestHandlerTimeoutMillis`
- if (this.requestQueue) {
- this.requestQueue.internalTimeoutMillis = this.internalTimeoutMillis;
- // for request queue v2, we want to lock requests for slightly longer than the request handler timeout so that there is some padding for locking-related overhead,
- // but never for less than a minute
- this.requestQueue.requestLockSecs = Math.max(this.requestHandlerTimeoutMillis / 1000 + 5, 60);
+ if (this.requestManager instanceof RequestProvider) {
+ this.applyRequestQueueTimeouts(this.requestManager);
}
this.maxRequestRetries = maxRequestRetries;
@@ -1171,9 +1174,9 @@ export class BasicCrawler<
private async createContextHelpers({ request, session }: { request: Request; session: ISession }) {
const enqueueLinksWrapper: CrawlingContext['enqueueLinks'] = async (options) => {
- const requestQueue = await this.getRequestQueue();
+ const requestManager = await this.getRequestManager();
- return await this.enqueueLinksWithCrawlDepth(options, request!, requestQueue);
+ return await this.enqueueLinksWithCrawlDepth(options, request!, requestManager);
};
const addRequests: CrawlingContext['addRequests'] = async (requests, options = {}) => {
const newCrawlDepth = request!.crawlDepth + 1;
@@ -1277,7 +1280,7 @@ export class BasicCrawler<
if (operationMode === 'ERROR') {
message = `Experiencing problems, ${failedDelta} failed requests in the past ${this.statusMessageLoggingInterval} seconds.`;
} else {
- const total = this.requestManager?.getTotalCount();
+ const total = await this.requestManager?.getTotalCount();
message = `Crawled ${this.stats.state.requestsFinished}${total ? `/${total}` : ''} pages, ${
this.stats.state.requestsFailed
} failed requests, desired concurrency ${this.autoscaledPool?.desiredConcurrency ?? 0}.`;
@@ -1324,14 +1327,15 @@ export class BasicCrawler<
// we need to purge the RQ to allow processing the same requests again — this is important so users can
// pass in failed requests back to the `crawler.run()`, otherwise they would be considered as handled and
// ignored — as a failed request is still handled.
- // By default (purgeRequestQueue unset or true), only the queue we created ourselves (ownedRequestQueue) is purged.
- // When `purgeRequestQueue` is explicitly `true`, we also purge a user-supplied queue.
+ // By default (purgeRequestQueue unset or true), only the manager we created ourselves (ownedRequestManager) is purged.
+ // When `purgeRequestQueue` is explicitly `true`, we also purge a user-supplied manager.
// When `purgeRequestQueue` is explicitly `false`, nothing is purged.
const shouldPurge = purgeRequestQueue !== false;
- const queueToPurge = this.ownedRequestQueue ?? (purgeRequestQueue === true ? this.requestQueue : undefined);
+ const managerToPurge =
+ this.ownedRequestManager ?? (purgeRequestQueue === true ? this.requestManager : undefined);
- if (queueToPurge && shouldPurge) {
- await queueToPurge.purge();
+ if (managerToPurge?.purge && shouldPurge) {
+ await managerToPurge.purge();
this.handledRequestsCount = 0; // This would've been reset by this._init() further down below, but at that point `handledRequestsCount` could prevent `addRequests` from adding the initial requests
}
@@ -1455,27 +1459,46 @@ export class BasicCrawler<
this.unexpectedStop = true;
}
- async getRequestQueue(): Promise<RequestProvider> {
- if (!this.requestQueue && this.requestList) {
- this.log.warningOnce(
- 'When using RequestList and RequestQueue at the same time, you should instantiate both explicitly and provide them in the crawler options, to ensure correctly handled restarts of the crawler.',
- );
+ /**
+ * Returns the crawler's {@apilink IRequestManager|request manager}, opening the default {@apilink RequestQueue}
+ * if none has been configured or opened yet.
+ */
+ async getRequestManager(): Promise<IRequestManager> {
+ if (!this.requestManager) {
+ this.requestManager = await this.openOwnedRequestQueue();
}
- if (!this.requestQueue) {
- this.requestQueue = await this._getRequestQueue();
- this.ownedRequestQueue = this.requestQueue;
- this.requestManager = undefined;
- }
+ return this.requestManager;
+ }
- if (!this.requestManager) {
- this.requestManager =
- this.requestList === undefined
- ? this.requestQueue
- : new RequestManagerTandem(this.requestList, this.requestQueue);
- }
+ /**
+ * @deprecated Use {@apilink BasicCrawler.getRequestManager|`getRequestManager()`} instead. This returns the
+ * crawler's request manager, which is no longer guaranteed to be a {@apilink RequestProvider}.
+ */
+ async getRequestQueue(): Promise<IRequestManager> {
+ return this.getRequestManager();
+ }
- return this.requestQueue;
+ /**
+ * Opens the default {@apilink RequestQueue}, applies the crawler's internal timeouts and records it as the
+ * crawler-owned manager (so it gets purged between repeated `run()` calls).
+ * @private
+ */
+ private async openOwnedRequestQueue(): Promise<IRequestManager> {
+ const requestQueue = await this._getRequestQueue();
+ this.applyRequestQueueTimeouts(requestQueue);
+ this.ownedRequestManager = requestQueue;
+ return requestQueue;
+ }
+
+ /**
+ * Overrides the default internal timeouts of a {@apilink RequestProvider} to respect `requestHandlerTimeoutMillis`.
+ */
+ private applyRequestQueueTimeouts(requestQueue: RequestProvider): void {
+ requestQueue.internalTimeoutMillis = this.internalTimeoutMillis;
+ // for request queue v2, we want to lock requests for slightly longer than the request handler timeout so that
+ // there is some padding for locking-related overhead, but never for less than a minute
+ requestQueue.requestLockSecs = Math.max(this.requestHandlerTimeoutMillis / 1000 + 5, 60);
}
/**
@@ -1547,18 +1570,18 @@ export class BasicCrawler<
return kvs.getAutoSavedValue<State>(BasicCrawler.CRAWLEE_STATE_KEY, defaultValue);
}
- protected get pendingRequestCountApproximation(): number {
- return this.requestManager?.getPendingCount() ?? 0;
+ protected async getPendingRequestCountApproximation(): Promise<number> {
+ return (await this.requestManager?.getPendingCount()) ?? 0;
}
- protected calculateEnqueuedRequestLimit(explicitLimit?: number): number | undefined {
+ protected async calculateEnqueuedRequestLimit(explicitLimit?: number): Promise<number | undefined> {
if (this.maxRequestsPerCrawl === undefined) {
return explicitLimit;
}
const limit = Math.max(
0,
- this.maxRequestsPerCrawl - this.handledRequestsCount - this.pendingRequestCountApproximation,
+ this.maxRequestsPerCrawl - this.handledRequestsCount - (await this.getPendingRequestCountApproximation()),
);
return Math.min(limit, explicitLimit ?? Infinity);
@@ -1605,9 +1628,9 @@ export class BasicCrawler<
requests: ReadonlyDeep<RequestsLike>,
options: CrawlerAddRequestsOptions = {},
): Promise<CrawlerAddRequestsResult> {
- await this.getRequestQueue();
+ await this.getRequestManager();
- const requestLimit = this.calculateEnqueuedRequestLimit();
+ const requestLimit = await this.calculateEnqueuedRequestLimit();
const skippedBecauseOfRobots = new Set<string>();
const skippedBecauseOfMaxCrawlDepth = new Set<string>();
@@ -1780,7 +1803,8 @@ export class BasicCrawler<
// (otherwise there would be no way)
this.autoscaledPool = new AutoscaledPool(this.autoscaledPoolOptions);
- await this.initializeRequestManager();
+ await this.getRequestManager();
+ await this._loadHandledRequestCount();
}
protected async runRequestHandler(crawlingContext: ExtendedContext): Promise<void> {
@@ -1851,10 +1875,12 @@ export class BasicCrawler<
});
}
- const requestListPersistPromise = (async () => {
- if (this.requestList) {
- if (await this.requestList.isFinished()) return;
- await this.requestList.persistState().catch((err) => {
+ const requestManagerPersistPromise = (async () => {
+ // The request manager persists its read-only loader's state, if it has one that supports persistence
+ // (e.g. a tandem wrapping a `RequestList`). For a plain `RequestQueue`, this is a no-op.
+ if (this.requestManager?.persistState) {
+ if (await this.requestManager.isFinished()) return;
+ await this.requestManager.persistState().catch((err) => {
if (err.message.includes('Cannot persist state.')) {
this.log.error(
"The crawler attempted to persist its request list's state and failed due to missing or " +
@@ -1872,30 +1898,7 @@ export class BasicCrawler<
}
})();
- await Promise.all([requestListPersistPromise, this.stats.persistState()]);
- }
-
- /**
- * Initializes the RequestManager based on the configured requestList and requestQueue.
- */
- private async initializeRequestManager() {
- if (this.requestManager !== undefined) {
- return;
- }
-
- if (this.requestList && this.requestQueue) {
- // Create a RequestManagerTandem if both RequestList and RequestQueue are provided
- this.requestManager = new RequestManagerTandem(this.requestList, this.requestQueue);
- } else if (this.requestQueue) {
- // Use RequestQueue directly if only it is provided
- this.requestManager = this.requestQueue;
- } else if (this.requestList) {
- // Use RequestList directly if only it is provided
- // Make it compatible with the IRequestManager interface
- this.requestManager = new RequestListAdapter(this.requestList);
- }
-
- // If neither RequestList nor RequestQueue is provided, leave the requestManager uninitialized until `getRequestQueue` is called
+ await Promise.all([requestManagerPersistPromise, this.stats.persistState()]);
}
/**
@@ -1914,7 +1917,7 @@ export class BasicCrawler<
* adding it back to the queue after the timeout passes. Returns `true` if the request
* should be ignored and will be reclaimed to the queue once ready.
*/
- protected delayRequest(request: Request, source: IRequestList | RequestProvider | IRequestManager) {
+ protected delayRequest(request: Request, source: RequestProvider | IRequestManager) {
const domain = getDomain(request.url);
if (!domain || !request) {
@@ -2041,7 +2044,7 @@ export class BasicCrawler<
protected async enqueueLinksWithCrawlDepth(
options: SetRequired<EnqueueLinksOptions, 'urls'>,
request: Request<Dictionary>,
- requestQueue: RequestProvider,
+ requestManager: IRequestManager,
): Promise<BatchAddRequestsResult> {
const transformRequestFunctionWrapper: RequestTransform = (requestOptions) => {
requestOptions.crawlDepth = request.crawlDepth + 1;
@@ -2061,7 +2064,7 @@ export class BasicCrawler<
: requestOptions;
};
- const limit = this.calculateEnqueuedRequestLimit(options.limit);
+ const limit = await this.calculateEnqueuedRequestLimit(options.limit);
// Create a request-scoped callback that logs enqueueLimit once per request handler call
// Only log if an explicit limit was passed to enqueueLinks (not the internal maxRequestsPerCrawl-derived limit)
@@ -2091,7 +2094,7 @@ export class BasicCrawler<
// The options below are merged with the user options, so an explicitly `undefined` value
// (e.g. `enqueueLinks({ urls, limit: config.limit })`) cannot discard the crawler defaults ⤵
- requestQueue: options.requestQueue ?? requestQueue,
+ requestManager: options.requestManager ?? requestManager,
robotsTxtFile: options.robotsTxtFile ?? (await this.getRobotsTxtFileForUrl(request.url)),
respectRobotsTxtFile: options.respectRobotsTxtFile ?? this.respectRobotsTxtFile,
onSkippedRequest,
@@ -2180,7 +2183,7 @@ export class BasicCrawler<
error: Error,
crawlingContext: CrawlingContext,
request: Request,
- source: IRequestList | IRequestManager,
+ source: IRequestManager,
): Promise<void> {
request.pushErrorMessage(error);
@@ -2312,7 +2315,7 @@ export class BasicCrawler<
*/
protected async _loadHandledRequestCount(): Promise<void> {
if (this.requestManager) {
- this.handledRequestsCount = await this.requestManager.handledCount();
+ this.handledRequestsCount = await this.requestManager.getHandledCount();
}
}
diff --git a/packages/browser-crawler/src/internals/browser-crawler.ts b/packages/browser-crawler/src/internals/browser-crawler.ts
index 2a49d1ea4..7997c3deb 100644
--- a/packages/browser-crawler/src/internals/browser-crawler.ts
+++ b/packages/browser-crawler/src/internals/browser-crawler.ts
@@ -7,10 +7,10 @@ import type {
Dictionary,
EnqueueLinksOptions,
ErrorHandler,
+ IRequestManager,
LoadedRequest,
Request,
RequestHandler,
- RequestProvider,
SkippedRequestCallback,
} from '@crawlee/basic';
import {
@@ -275,15 +275,18 @@ export interface BrowserCrawlerOptions<
* If the target website doesn't need JavaScript, we should consider using the {@apilink CheerioCrawler},
* which downloads the pages using raw HTTP requests and is about 10x faster.
*
- * The source URLs are represented by the {@apilink Request} objects that are fed from the {@apilink RequestList} or {@apilink RequestQueue} instances
- * provided by the {@apilink BrowserCrawlerOptions.requestList|`requestList`} or {@apilink BrowserCrawlerOptions.requestQueue|`requestQueue`}
- * constructor options, respectively. If neither `requestList` nor `requestQueue` options are provided,
+ * The source URLs are represented by the {@apilink Request} objects that are fed from the
+ * {@apilink IRequestManager|request manager} provided via the {@apilink BrowserCrawlerOptions.requestManager|`requestManager`}
+ * constructor option (a {@apilink RequestQueue} is itself a request manager). If no `requestManager` is provided,
* the crawler will open the default request queue either when the {@apilink BrowserCrawler.addRequests|`crawler.addRequests()`} function is called,
* or if `requests` parameter (representing the initial requests) of the {@apilink BrowserCrawler.run|`crawler.run()`} function is provided.
*
- * If both {@apilink BrowserCrawlerOptions.requestList|`requestList`} and {@apilink BrowserCrawlerOptions.requestQueue|`requestQueue`} options are used,
- * the instance first processes URLs from the {@apilink RequestList} and automatically enqueues all of them
- * to the {@apilink RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
+ * To read from a read-only source such as a {@apilink RequestList} while still being able to enqueue new requests,
+ * combine it with a queue into a {@apilink RequestManagerTandem} via {@apilink IRequestLoader.toTandem|`requestLoader.toTandem()`}
+ * and pass the result as `requestManager`.
+ *
+ * > The {@apilink BrowserCrawlerOptions.requestList|`requestList`} and {@apilink BrowserCrawlerOptions.requestQueue|`requestQueue`}
+ * > options are deprecated; they are still accepted and folded into a single `requestManager` for back-compat.
*
* The crawler finishes when there are no more {@apilink Request} objects to crawl.
*
@@ -550,7 +553,7 @@ export abstract class BrowserCrawler<
// pre-clamping it here would make the crawler log the internal limit as a user-provided one
options: enqueueOptions,
page,
- requestQueue: await this.getRequestQueue(),
+ requestManager: await this.getRequestManager(),
robotsTxtFile: await this.getRobotsTxtFileForUrl(crawlingContext.request.url),
onSkippedRequest: this.handleSkippedRequest,
originalRequestUrl: crawlingContext.request.url,
@@ -753,9 +756,9 @@ export abstract class BrowserCrawler<
/** @internal */
interface EnqueueLinksInternalOptions {
- options?: ReadonlyDeep<Omit<EnqueueLinksOptions, 'requestQueue'>> & Pick<EnqueueLinksOptions, 'requestQueue'>;
+ options?: ReadonlyDeep<Omit<EnqueueLinksOptions, 'requestManager'>> & Pick<EnqueueLinksOptions, 'requestManager'>;
page: CommonPage;
- requestQueue: RequestProvider;
+ requestManager: IRequestManager;
robotsTxtFile?: RobotsTxtFile;
onSkippedRequest?: SkippedRequestCallback;
originalRequestUrl: string;
@@ -765,7 +768,7 @@ interface EnqueueLinksInternalOptions {
/** @internal */
interface BoundEnqueueLinksInternalOptions {
enqueueLinks: BasicCrawlingContext['enqueueLinks'];
- options?: ReadonlyDeep<Omit<EnqueueLinksOptions, 'requestQueue'>> & Pick<EnqueueLinksOptions, 'requestQueue'>;
+ options?: ReadonlyDeep<Omit<EnqueueLinksOptions, 'requestManager'>> & Pick<EnqueueLinksOptions, 'requestManager'>;
originalRequestUrl: string;
finalRequestUrl?: string;
page: CommonPage;
@@ -806,7 +809,7 @@ export async function browserCrawlerEnqueueLinks(
}
return enqueueLinks({
- requestQueue: options.requestQueue,
+ requestManager: options.requestManager,
robotsTxtFile: options.robotsTxtFile,
onSkippedRequest: options.onSkippedRequest,
urls,
diff --git a/packages/cheerio-crawler/src/internals/cheerio-crawler.ts b/packages/cheerio-crawler/src/internals/cheerio-crawler.ts
index 49bf74a89..342350afe 100644
--- a/packages/cheerio-crawler/src/internals/cheerio-crawler.ts
+++ b/packages/cheerio-crawler/src/internals/cheerio-crawler.ts
@@ -6,8 +6,8 @@ import type {
HttpCrawlerOptions,
InternalHttpCrawlingContext,
InternalHttpHook,
+ IRequestManager,
RequestHandler,
- RequestProvider,
RouterHandler,
RouterRoutes,
RouteSchemas,
@@ -116,13 +116,15 @@ export type CheerioRequestHandler<
* and then invokes the user-provided {@apilink CheerioCrawlerOptions.requestHandler} to extract page data
* using a [jQuery](https://jquery.com/)-like interface to the parsed HTML DOM.
*
- * The source URLs are represented using {@apilink Request} objects that are fed from
- * {@apilink RequestList} or {@apilink RequestQueue} instances provided by the {@apilink CheerioCrawlerOptions.requestList}
- * or {@apilink CheerioCrawlerOptions.requestQueue} constructor options, respectively.
+ * The source URLs are represented using {@apilink Request} objects that are fed from the
+ * {@apilink IRequestManager|request manager} provided via the {@apilink CheerioCrawlerOptions.requestManager|`requestManager`}
+ * constructor option (a {@apilink RequestQueue} is itself a request manager). To read from a read-only source such
+ * as a {@apilink RequestList} while still being able to enqueue new requests, combine it with a queue into a
+ * {@apilink RequestManagerTandem} via {@apilink IRequestLoader.toTandem|`requestLoader.toTandem()`} and pass the
+ * result as `requestManager`.
*
- * If both {@apilink CheerioCrawlerOptions.requestList} and {@apilink CheerioCrawlerOptions.requestQueue} are used,
- * the instance first processes URLs from the {@apilink RequestList} and automatically enqueues all of them
- * to {@apilink RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
+ * > The {@apilink CheerioCrawlerOptions.requestList|`requestList`} and {@apilink CheerioCrawlerOptions.requestQueue|`requestQueue`}
+ * > options are deprecated; they are still accepted and folded into a single `requestManager` for back-compat.
*
* The crawler finishes when there are no more {@apilink Request} objects to crawl.
*
@@ -249,7 +251,7 @@ export class CheerioCrawler<
// pre-clamping it here would make the crawler log the internal limit as a user-provided one
options: enqueueOptions,
$: crawlingContext.$,
- requestQueue: await this.getRequestQueue(),
+ requestManager: await this.getRequestManager(),
robotsTxtFile: await this.getRobotsTxtFileForUrl(crawlingContext.request.url),
onSkippedRequest: this.handleSkippedRequest,
originalRequestUrl: crawlingContext.request.url,
@@ -276,7 +278,7 @@ export class CheerioCrawler<
interface EnqueueLinksInternalOptions {
options?: EnqueueLinksOptions;
$: cheerio.CheerioAPI | null;
- requestQueue: RequestProvider;
+ requestManager: IRequestManager;
robotsTxtFile?: RobotsTxtFile;
onSkippedRequest?: SkippedRequestCallback;
originalRequestUrl: string;
@@ -328,7 +330,7 @@ export async function cheerioCrawlerEnqueueLinks(
});
}
return enqueueLinks({
- requestQueue: options.requestQueue,
+ requestManager: options.requestManager,
robotsTxtFile: options.robotsTxtFile,
onSkippedRequest: options.onSkippedRequest,
urls,
diff --git a/packages/core/src/crawlers/crawler_commons.ts b/packages/core/src/crawlers/crawler_commons.ts
index 6c95d24c3..3b071d294 100644
--- a/packages/core/src/crawlers/crawler_commons.ts
+++ b/packages/core/src/crawlers/crawler_commons.ts
@@ -78,7 +78,7 @@ export interface RestrictedCrawlingContext<UserData extends Dictionary = Diction
* @param [options] All `enqueueLinks()` parameters are passed via an options object.
*/
enqueueLinks: (
- options: ReadonlyDeep<Omit<SetRequired<EnqueueLinksOptions, 'urls'>, 'requestQueue' | 'robotsTxtFile'>>,
+ options: ReadonlyDeep<Omit<SetRequired<EnqueueLinksOptions, 'urls'>, 'requestManager' | 'robotsTxtFile'>>,
) => Promise<unknown>;
/**
@@ -137,8 +137,8 @@ export interface CrawlingContext<UserData extends Dictionary = Dictionary> exten
* @returns Promise that resolves to {@apilink BatchAddRequestsResult} object.
*/
enqueueLinks(
- options: ReadonlyDeep<Omit<SetRequired<EnqueueLinksOptions, 'urls'>, 'requestQueue' | 'robotsTxtFile'>> &
- Pick<EnqueueLinksOptions, 'requestQueue' | 'robotsTxtFile'>,
+ options: ReadonlyDeep<Omit<SetRequired<EnqueueLinksOptions, 'urls'>, 'requestManager' | 'robotsTxtFile'>> &
+ Pick<EnqueueLinksOptions, 'requestManager' | 'robotsTxtFile'>,
): Promise<unknown>;
/**
diff --git a/packages/core/src/enqueue_links/enqueue_links.ts b/packages/core/src/enqueue_links/enqueue_links.ts
index d3d0c065c..2e70a037b 100644
--- a/packages/core/src/enqueue_links/enqueue_links.ts
+++ b/packages/core/src/enqueue_links/enqueue_links.ts
@@ -7,10 +7,10 @@ import type { SetRequired } from 'type-fest';
import type { RequestOptions } from '../request.js';
import { Request } from '../request.js';
import { serviceLocator } from '../service_locator.js';
+import type { IRequestManager } from '../storages/request_manager.js';
import type {
AddRequestsBatchedOptions,
AddRequestsBatchedResult,
- RequestProvider,
RequestQueueOperationOptions,
} from '../storages/request_provider.js';
import type {
@@ -44,8 +44,8 @@ export interface EnqueueLinksOptions extends RequestQueueOperationOptions {
/** An array of URLs to enqueue. */
urls?: readonly string[];
- /** A request queue to which the URLs will be enqueued. */
- requestQueue?: RequestProvider;
+ /** A request manager to which the URLs will be enqueued. */
+ requestManager?: IRequestManager;
/** A CSS selector matching links to be enqueued. */
selector?: string;
@@ -228,7 +228,7 @@ export interface EnqueueLinksOptions extends RequestQueueOperationOptions {
* ```javascript
* await enqueueLinks({
* urls: aListOfFoundUrls,
- * requestQueue,
+ * requestManager,
* selector: 'a.product-detail',
* globs: [
* 'https://www.example.com/handbags/*',
@@ -241,8 +241,8 @@ export interface EnqueueLinksOptions extends RequestQueueOperationOptions {
* @returns Promise that resolves to {@apilink BatchAddRequestsResult} object.
*/
export async function enqueueLinks(
- options: SetRequired<Omit<EnqueueLinksOptions, 'requestQueue'>, 'urls'> & {
- requestQueue: {
+ options: SetRequired<Omit<EnqueueLinksOptions, 'requestManager'>, 'urls'> & {
+ requestManager: {
addRequestsBatched: (
requests: Request<Dictionary>[],
options: AddRequestsBatchedOptions,
@@ -263,7 +263,7 @@ export async function enqueueLinks(
options as any,
ow.object.exactShape({
urls: ow.array.ofType(ow.string),
- requestQueue: ow.object.hasKeys('addRequestsBatched'),
+ requestManager: ow.object.hasKeys('addRequestsBatched'),
robotsTxtFile: ow.optional.object.hasKeys('isAllowed'),
respectRobotsTxtFile: ow.optional.any(ow.boolean, ow.object.exactShape({ userAgent: ow.optional.string })),
onSkippedRequest: ow.optional.function,
@@ -288,7 +288,7 @@ export async function enqueueLinks(
);
const {
- requestQueue,
+ requestManager,
limit,
urls,
// oxlint-disable-next-line typescript/no-deprecated -- still accepted for backwards compat
@@ -469,7 +469,7 @@ export async function enqueueLinks(
requests = requests.slice(0, limit);
}
- const { addedRequests, requestsOverLimit } = await requestQueue.addRequestsBatched(requests, {
+ const { addedRequests, requestsOverLimit } = await requestManager.addRequestsBatched(requests, {
forefront,
waitForAllRequestsToBeAdded,
maxNewRequests: limit,
diff --git a/packages/core/src/storages/index.ts b/packages/core/src/storages/index.ts
index 9bc71874d..cf66ee667 100644
--- a/packages/core/src/storages/index.ts
+++ b/packages/core/src/storages/index.ts
@@ -1,7 +1,8 @@
export * from './dataset.js';
export * from './key_value_store.js';
export * from './request_list.js';
-export * from './request_list_adapter.js';
+export type * from './request_loader.js';
+export type * from './request_manager.js';
export * from './request_provider.js';
export { RequestQueueV1 } from './request_queue.js';
export { RequestQueue } from './request_queue_v2.js';
@@ -9,5 +10,5 @@ export { RequestQueue as RequestQueueV2 } from './request_queue_v2.js';
export * from './storage_instance_manager.js';
export * from './utils.js';
export * from './access_checking.js';
-export * from './sitemap_request_list.js';
+export * from './sitemap_request_loader.js';
export * from './request_manager_tandem.js';
diff --git a/packages/core/src/storages/request_list.ts b/packages/core/src/storages/request_list.ts
index 6d5471e0b..02ac29b4a 100644
--- a/packages/core/src/storages/request_list.ts
+++ b/packages/core/src/storages/request_list.ts
@@ -10,6 +10,8 @@ import { type InternalSource, Request, type RequestOptions, type Source } from '
import { createDeserialize, serializeArray } from '../serialization.js';
import { serviceLocator } from '../service_locator.js';
import { KeyValueStore } from './key_value_store.js';
+import type { IRequestLoader } from './request_loader.js';
+import type { IRequestManager } from './request_manager.js';
import { purgeDefaultStorages } from './utils.js';
/** @internal */
@@ -20,74 +22,6 @@ export const REQUESTS_PERSISTENCE_KEY = 'REQUEST_LIST_REQUESTS';
const CONTENT_TYPE_BINARY = 'application/octet-stream';
-/**
- * Represents a static list of URLs to crawl.
- */
-export interface IRequestList {
- /**
- * Returns the total number of unique requests present in the list.
- */
- length(): number;
-
- /**
- * Returns `true` if all requests were already handled and there are no more left.
- */
- isFinished(): Promise<boolean>;
-
- /**
- * Resolves to `true` if the next call to {@apilink IRequestList.fetchNextRequest} function
- * would return `null`, otherwise it resolves to `false`.
- * Note that even if the list is empty, there might be some pending requests currently being processed.
- */
- isEmpty(): Promise<boolean>;
-
- /**
- * Returns number of handled requests.
- */
- handledCount(): number;
-
- /**
- * Persists the current state of the `IRequestList` into the default {@apilink KeyValueStore}.
- * The state is persisted automatically in regular intervals, but calling this method manually
- * is useful in cases where you want to have the most current state available after you pause
- * or stop fetching its requests. For example after you pause or abort a crawl. Or just before
- * a server migration.
- */
- persistState(): Promise<void>;
-
- /**
- * Gets the next {@apilink Request} to process. First, the function gets a request previously reclaimed
- * using the {@apilink RequestList.reclaimRequest} function, if there is any.
- * Otherwise it gets the next request from sources.
- *
- * The function's `Promise` resolves to `null` if there are no more
- * requests to process.
- */
- fetchNextRequest(): Promise<Request | null>;
-
- /**
- * Can be used to iterate over the `RequestList` instance in a `for await .. of` loop.
- * Provides an alternative for the repeated use of `fetchNextRequest`.
- */
- [Symbol.asyncIterator](): AsyncGenerator<Request>;
-
- /**
- * Reclaims request to the list if its processing failed.
- * The request will become available in the next `this.fetchNextRequest()`.
- */
- reclaimRequest(request: Request): Promise<void>;
-
- /**
- * Marks request as handled after successful processing.
- */
- markRequestHandled(request: Request): Promise<void>;
-
- /**
- * @internal
- */
- inProgress: Set<string>;
-}
-
export interface RequestListOptions {
/**
* An array of sources of URLs for the {@apilink RequestList}. It can be either an array of strings,
@@ -265,8 +199,8 @@ export interface RequestListOptions {
* > In practical terms, such a combination can be useful when there is a large number of initial URLs,
* > but more URLs would be added dynamically by the crawler.
*
- * `RequestList` has an internal state where it stores information about which requests were already handled,
- * which are in progress and which were reclaimed. The state may be automatically persisted to the default
+ * `RequestList` has an internal state where it stores information about which requests were already handled
+ * and which are in progress. The state may be automatically persisted to the default
* {@apilink KeyValueStore} by setting the `persistStateKey` option so that if the Node.js process is restarted,
* the crawling can continue where it left off. The automated persisting is launched upon receiving the `persistState`
* event that is periodically emitted by {@apilink EventManager}.
@@ -304,7 +238,7 @@ export interface RequestListOptions {
* ```
* @category Sources
*/
-export class RequestList implements IRequestList {
+export class RequestList implements IRequestLoader {
private log: CrawleeLogger = serviceLocator.getLogger().child({ prefix: 'RequestList' });
/**
@@ -327,10 +261,11 @@ export class RequestList implements IRequestList {
inProgress = new Set<string>();
/**
- * Set of `uniqueKey`s of requests for which reclaimRequest() was called.
+ * `uniqueKey`s of requests that were in progress when the state was last persisted and thus need to be
+ * re-crawled after a restart. They are served before advancing through the rest of the sources.
* @internal
*/
- reclaimed = new Set<string>();
+ private requestsToRetry: string[] = [];
/**
* Starts as true because until we handle the first request, the list is effectively persisted by doing nothing.
@@ -619,8 +554,8 @@ export class RequestList implements IRequestList {
}
}
- // All in-progress requests need to be re-crawled
- this.reclaimed = new Set(this.inProgress);
+ // All in-progress requests were interrupted and need to be re-crawled.
+ this.requestsToRetry = [...this.inProgress];
}
/**
@@ -667,7 +602,7 @@ export class RequestList implements IRequestList {
async isEmpty(): Promise<boolean> {
this._ensureIsInitialized();
- return this.reclaimed.size === 0 && this.nextIndex >= this.requests.length;
+ return this.requestsToRetry.length === 0 && this.nextIndex >= this.requests.length;
}
/**
@@ -685,10 +620,9 @@ export class RequestList implements IRequestList {
async fetchNextRequest(): Promise<Request | null> {
this._ensureIsInitialized();
- // First return reclaimed requests if any.
- const uniqueKey = this.reclaimed.values().next().value;
+ // First re-serve any requests that were interrupted before the last state persist.
+ const uniqueKey = this.requestsToRetry.shift();
if (uniqueKey) {
- this.reclaimed.delete(uniqueKey);
const index = this.uniqueKeyToIndex[uniqueKey];
return this.ensureRequest(this.requests[index], index);
}
@@ -733,26 +667,13 @@ export class RequestList implements IRequestList {
const { uniqueKey } = request;
this._ensureUniqueKeyValid(uniqueKey);
- this._ensureInProgressAndNotReclaimed(uniqueKey);
+ this._ensureInProgress(uniqueKey);
this._ensureIsInitialized();
this.inProgress.delete(uniqueKey);
this.isStatePersisted = false;
}
- /**
- * @inheritDoc
- */
- async reclaimRequest(request: Request): Promise<void> {
- const { uniqueKey } = request;
-
- this._ensureUniqueKeyValid(uniqueKey);
- this._ensureInProgressAndNotReclaimed(uniqueKey);
- this._ensureIsInitialized();
-
- this.reclaimed.add(uniqueKey);
- }
-
/**
* Adds all fetched requests from a URL from a remote resource.
*/
@@ -861,15 +782,12 @@ export class RequestList implements IRequestList {
}
/**
- * Checks that request is not reclaimed and throws an error if so.
+ * Checks that a request is currently being processed and throws an error if not.
*/
- protected _ensureInProgressAndNotReclaimed(uniqueKey: string): void {
+ protected _ensureInProgress(uniqueKey: string): void {
if (!this.inProgress.has(uniqueKey)) {
throw new Error(`The request is not being processed (uniqueKey: ${uniqueKey})`);
}
- if (this.reclaimed.has(uniqueKey)) {
- throw new Error(`The request was already reclaimed (uniqueKey: ${uniqueKey})`);
- }
}
/**
@@ -886,16 +804,38 @@ export class RequestList implements IRequestList {
/**
* Returns the total number of unique requests present in the `RequestList`.
*/
- length(): number {
+ async getTotalCount(): Promise<number> {
this._ensureIsInitialized();
return this.requests.length;
}
+ /**
+ * Returns the number of pending requests in the `RequestList`.
+ */
+ async getPendingCount(): Promise<number> {
+ this._ensureIsInitialized();
+
+ return this.requests.length - (this.nextIndex - this.inProgress.size);
+ }
+
+ /**
+ * Combines this list with a request manager (a {@apilink RequestQueue} by default) into a
+ * {@apilink RequestManagerTandem}, allowing requests to be added and reclaimed while still
+ * being read from this list first.
+ */
+ async toTandem(requestManager?: IRequestManager): Promise<IRequestManager> {
+ // Import here to avoid circular imports.
+ const { RequestManagerTandem } = await import('./request_manager_tandem.js');
+ const { RequestQueue } = await import('./request_queue_v2.js');
+
+ return new RequestManagerTandem(this, requestManager ?? (await RequestQueue.open()));
+ }
+
/**
* @inheritDoc
*/
- handledCount(): number {
+ async getHandledCount(): Promise<number> {
this._ensureIsInitialized();
return this.nextIndex - this.inProgress.size;
diff --git a/packages/core/src/storages/request_list_adapter.ts b/packages/core/src/storages/request_list_adapter.ts
deleted file mode 100644
index 363622955..000000000
--- a/packages/core/src/storages/request_list_adapter.ts
+++ /dev/null
@@ -1,102 +0,0 @@
-import type { Dictionary } from '@crawlee/types';
-
-import type { Request } from '../request.js';
-import type { IRequestList } from './request_list.js';
-import type {
- AddRequestsBatchedResult,
- IRequestManager,
- RequestQueueOperationInfo,
- RequestQueueOperationOptions,
-} from './request_provider.js';
-
-/**
- * Adapts the IRequestList interface to the IRequestManager interface.
- * It simply throws an exception when inserting requests is attempted.
- * @internal
- */
-export class RequestListAdapter implements IRequestManager {
- constructor(private requestList: IRequestList) {}
-
- /**
- * @inheritdoc
- */
- async isFinished(): Promise<boolean> {
- return this.requestList.isFinished();
- }
-
- /**
- * @inheritdoc
- */
- async isEmpty(): Promise<boolean> {
- return this.requestList.isEmpty();
- }
-
- /**
- * @inheritdoc
- */
- async handledCount(): Promise<number> {
- return Promise.resolve(this.requestList.handledCount());
- }
-
- /**
- * @inheritdoc
- */
- getTotalCount(): number {
- return this.requestList.length();
- }
-
- /**
- * @inheritdoc
- */
- getPendingCount(): number {
- return this.requestList.length() - this.requestList.handledCount();
- }
-
- /**
- * @inheritdoc
- */
- async fetchNextRequest<T extends Dictionary = Dictionary>(): Promise<Request<T> | null> {
- return await this.requestList.fetchNextRequest();
- }
-
- /**
- * @inheritdoc
- */
- async markRequestHandled(request: Request): Promise<void> {
- return this.requestList.markRequestHandled(request);
- }
-
- /**
- * @inheritdoc
- */
- async reclaimRequest(
- request: Request,
- _options?: RequestQueueOperationOptions,
- ): Promise<RequestQueueOperationInfo | null> {
- await this.requestList.reclaimRequest(request);
- return null;
- }
-
- /**
- * @inheritdoc
- */
- async *[Symbol.asyncIterator]() {
- for await (const request of this.requestList) {
- yield request;
- }
- }
-
- /**
- * @inheritdoc
- */
- addRequestsBatched(): Promise<AddRequestsBatchedResult> {
- throw new Error('Cannot add requests to a read-only request list');
- }
-
- /**
- * @inheritdoc
- */
- addRequest(): Promise<RequestQueueOperationInfo> {
- throw new Error('Cannot add requests to a read-only request list');
- }
-}
diff --git a/packages/core/src/storages/request_loader.ts b/packages/core/src/storages/request_loader.ts
new file mode 100644
index 000000000..9292860f8
--- /dev/null
+++ b/packages/core/src/storages/request_loader.ts
@@ -0,0 +1,107 @@
+import type { Dictionary } from '@crawlee/types';
+
+import type { Request } from '../request.js';
+import type { IRequestManager } from './request_manager.js';
+import type { RequestQueueOperationInfo } from './request_provider.js';
+
+/**
+ * An abstract interface defining a read-only stream of requests to crawl.
+ *
+ * Request loaders are used to manage and provide access to a storage of crawling requests.
+ *
+ * Key responsibilities:
+ * - Fetching the next request to be processed.
+ * - Marking requests as handled once they are no longer in progress.
+ * - Managing state information such as the total and handled request counts.
+ *
+ * ## Request lifecycle contract
+ *
+ * Every request returned by {@apilink IRequestLoader.fetchNextRequest} is considered **in progress**
+ * until it is passed to {@apilink IRequestLoader.markRequestHandled}. Once you fetch a request, you are
+ * obligated to eventually mark it as handled — there is no way to hand a request back to a loader
+ * (only an {@apilink IRequestManager} can reclaim requests for a retry). "Handled" therefore means
+ * "finished with this request", whether processing succeeded or was abandoned after exhausting retries.
+ *
+ * Honoring this contract matters for three reasons:
+ * - **Restarts and migrations:** loaders that persist their state (see {@apilink IRequestLoader.persistState})
+ * treat in-progress requests as interrupted and re-serve them after a restart. A request that is fetched
+ * but never marked handled will be crawled again.
+ * - **Termination detection:** {@apilink IRequestLoader.isFinished} only resolves to `true` once nothing is
+ * in progress. Leaving a request unmarked keeps the crawler running indefinitely.
+ * - **Bookkeeping:** the handled and pending counts are derived from the set of in-progress requests, so
+ * skipping {@apilink IRequestLoader.markRequestHandled} corrupts {@apilink IRequestLoader.getHandledCount}
+ * and {@apilink IRequestLoader.getPendingCount}.
+ *
+ * Concrete implementations such as {@apilink RequestList} or {@apilink SitemapRequestLoader} build on this interface.
+ * The {@apilink IRequestManager} interface extends it with the capability to enqueue and reclaim requests.
+ */
+export interface IRequestLoader {
+ /**
+ * Returns an approximation of the total number of requests in the loader (i.e. pending + handled).
+ */
+ getTotalCount(): Promise<number>;
+
+ /**
+ * Returns an approximation of the number of pending requests in the loader.
+ */
+ getPendingCount(): Promise<number>;
+
+ /**
+ * Returns the number of requests in the loader that have been handled.
+ */
+ getHandledCount(): Promise<number>;
+
+ /**
+ * Returns `true` if all requests were already handled and there are no more left.
+ */
+ isFinished(): Promise<boolean>;
+
+ /**
+ * Resolves to `true` if the next call to {@apilink IRequestLoader.fetchNextRequest} function
+ * would return `null`, otherwise it resolves to `false`.
+ * Note that even if the loader is empty, there might be some pending requests currently being processed.
+ */
+ isEmpty(): Promise<boolean>;
+
+ /**
+ * Gets the next {@apilink Request} to process, or `null` if there are no more pending requests.
+ *
+ * The returned request is marked as **in progress** and remains so until it is passed to
+ * {@apilink IRequestLoader.markRequestHandled}. The caller is responsible for eventually marking
+ * every fetched request as handled; otherwise the loader never considers itself finished and the
+ * request may be re-served after a restart. See the request lifecycle contract on {@apilink IRequestLoader}.
+ */
+ fetchNextRequest<T extends Dictionary = Dictionary>(): Promise<Request<T> | null>;
+
+ /**
+ * Can be used to iterate over the loader instance in a `for await .. of` loop.
+ * Provides an alternative for the repeated use of `fetchNextRequest`.
+ */
+ [Symbol.asyncIterator](): AsyncGenerator<Request>;
+
+ /**
+ * Marks a request previously returned by {@apilink IRequestLoader.fetchNextRequest} as handled,
+ * removing it from the set of in-progress requests.
+ *
+ * Call this once you are done with the request — whether processing succeeded or was abandoned after
+ * exhausting retries. Because a loader cannot take a request back, marking it handled is the only way to
+ * signal completion; failing to do so prevents {@apilink IRequestLoader.isFinished} from ever resolving to
+ * `true` and skews the handled and pending counts. See the request lifecycle contract on {@apilink IRequestLoader}.
+ */
+ markRequestHandled(request: Request): Promise<RequestQueueOperationInfo | void | null>;
+
+ /**
+ * Persists the current state of the loader into the default {@apilink KeyValueStore}.
+ *
+ * Not all loaders support persistence; implementations that do not should leave this `undefined`.
+ */
+ persistState?(): Promise<void>;
+
+ /**
+ * Combines the loader with a request manager to support adding and reclaiming requests.
+ *
+ * @param requestManager Request manager to combine the loader with. If not provided, the default
+ * {@apilink RequestQueue} is used.
+ */
+ toTandem?(requestManager?: IRequestManager): Promise<IRequestManager>;
+}
diff --git a/packages/core/src/storages/request_manager.ts b/packages/core/src/storages/request_manager.ts
new file mode 100644
index 000000000..64cae64e7
--- /dev/null
+++ b/packages/core/src/storages/request_manager.ts
@@ -0,0 +1,34 @@
+import type { Request, Source } from '../request.js';
+import type { IRequestLoader } from './request_loader.js';
+import type {
+ AddRequestsBatchedOptions,
+ AddRequestsBatchedResult,
+ RequestQueueOperationInfo,
+ RequestQueueOperationOptions,
+} from './request_provider.js';
+
+export type RequestsLike = AsyncIterable<Source | string> | Iterable<Source | string> | (Source | string)[];
+
+/**
+ * Extends the read-only {@apilink IRequestLoader} interface with the capability to enqueue new requests
+ * and reclaim failed ones.
+ */
+export interface IRequestManager extends IRequestLoader {
+ /**
+ * Reclaims request to the provider if its processing failed.
+ * The request will be returned by some subsequent `fetchNextRequest()` call.
+ */
+ reclaimRequest(request: Request, options?: RequestQueueOperationOptions): Promise<RequestQueueOperationInfo | null>;
+
+ addRequest(requestLike: Source, options?: RequestQueueOperationOptions): Promise<RequestQueueOperationInfo>;
+
+ addRequestsBatched(requests: RequestsLike, options?: AddRequestsBatchedOptions): Promise<AddRequestsBatchedResult>;
+
+ /**
+ * Remove all requests from the queue but keep the queue itself, resetting it
+ * so it can be reused (e.g. across multiple `crawler.run()` calls).
+ *
+ * Implementations that do not support purging may leave this `undefined`.
+ */
+ purge?(): Promise<void>;
+}
diff --git a/packages/core/src/storages/request_manager_tandem.ts b/packages/core/src/storages/request_manager_tandem.ts
index 164bbba19..d294f0b27 100644
--- a/packages/core/src/storages/request_manager_tandem.ts
+++ b/packages/core/src/storages/request_manager_tandem.ts
@@ -3,85 +3,118 @@ import type { Dictionary } from '@crawlee/types';
import type { CrawleeLogger } from '../log.js';
import type { Request, Source } from '../request.js';
import { serviceLocator } from '../service_locator.js';
-import type { IRequestList } from './request_list.js';
+import type { IRequestLoader } from './request_loader.js';
+import type { IRequestManager, RequestsLike } from './request_manager.js';
import type {
AddRequestsBatchedOptions,
AddRequestsBatchedResult,
- IRequestManager,
RequestQueueOperationInfo,
RequestQueueOperationOptions,
- RequestsLike,
} from './request_provider.js';
/**
- * A request manager that combines a RequestList and a RequestQueue.
- * It first reads requests from the RequestList and then, when needed,
- * transfers them in batches to the RequestQueue.
+ * A request manager that combines a {@apilink IRequestLoader} (such as a `RequestList`) with a writable
+ * {@apilink IRequestManager} (such as a `RequestQueue`).
+ * It first reads requests from the loader and then, when needed, transfers them in batches to the manager.
*/
export class RequestManagerTandem implements IRequestManager {
private log: CrawleeLogger;
- private requestList: IRequestList;
- private requestQueue: IRequestManager;
+ private requestLoader: IRequestLoader;
+ private requestManagerPromise?: Promise<IRequestManager>;
+ private resolvedRequestManager?: IRequestManager;
- constructor(requestList: IRequestList, requestQueue: IRequestManager) {
+ private requestManagerFactory: () => IRequestManager | Promise<IRequestManager>;
+
+ /**
+ * @param requestLoader The read-only loader to read requests from first.
+ * @param requestManager The writable manager to transfer requests into and enqueue new ones. May be passed as a
+ * factory function so that the tandem can be constructed synchronously and the manager opened lazily on first use
+ * (e.g. a lazily-opened default {@apilink RequestQueue}).
+ */
+ constructor(
+ requestLoader: IRequestLoader,
+ requestManager: IRequestManager | (() => IRequestManager | Promise<IRequestManager>),
+ ) {
this.log = serviceLocator.getLogger().child({ prefix: 'RequestManagerTandem' });
- this.requestList = requestList;
- this.requestQueue = requestQueue;
+ this.requestLoader = requestLoader;
+ this.requestManagerFactory = typeof requestManager === 'function' ? requestManager : () => requestManager;
+ }
+
+ /**
+ * Resolves the writable request manager, opening it lazily (via the factory) on first use and memoizing the result.
+ * @private
+ */
+ private async getRequestManager(): Promise<IRequestManager> {
+ if (this.resolvedRequestManager === undefined) {
+ this.requestManagerPromise ??= Promise.resolve(this.requestManagerFactory());
+ this.resolvedRequestManager = await this.requestManagerPromise;
+ }
+ return this.resolvedRequestManager;
}
/**
- * Transfers a batch of requests from the RequestList to the RequestQueue.
- * Handles both successful transfers and failures appropriately.
+ * Transfers a single request from the read-only loader to the writable manager.
+ * If the transfer fails, the request is dropped (and logged) rather than reclaimed.
+ *
+ * @returns `true` if a request was successfully transferred (or there was nothing to transfer), and `false` if a
+ * transfer was attempted but failed - in which case the caller should not fetch from the manager this round.
* @private
*/
- private async transferNextBatchToQueue(): Promise<void> {
- const request = await this.requestList.fetchNextRequest();
+ private async transferNextRequestToQueue(): Promise<boolean> {
+ const request = await this.requestLoader.fetchNextRequest();
if (request === null) {
- return;
+ return true;
}
+ const requestManager = await this.getRequestManager();
+
try {
- await this.requestQueue.addRequest(request, { forefront: true });
+ await requestManager.addRequest(request, { forefront: true });
+ return true;
} catch (error) {
- // If requestQueue.addRequest() fails here then we must reclaim it back to
- // the RequestList because probably it's not yet in the queue!
- this.log.error(
- 'Adding of request from the RequestList to the RequestQueue failed, reclaiming request back to the list.',
- { request },
+ this.log.exception(
+ error as Error,
+ 'Adding request from the RequestLoader to the RequestManager failed, the request has been dropped.',
+ { url: request.url, uniqueKey: request.uniqueKey },
);
- await this.requestList.reclaimRequest(request);
- return;
+ return false;
+ } finally {
+ // Mark it as handled so that the request doesn't get stuck in the `inProgress` state in the loader.
+ await this.requestLoader.markRequestHandled(request);
}
-
- await this.requestList.markRequestHandled(request);
}
/**
- * Fetches the next request from the RequestQueue. If the queue is empty and the RequestList
- * is not finished, it will transfer a batch of requests from the list to the queue first.
+ * Fetches the next request from the request manager. If the manager is empty and the loader
+ * is not finished, it will transfer a request from the loader to the manager first.
* @inheritdoc
*/
async fetchNextRequest<T extends Dictionary = Dictionary>(): Promise<Request<T> | null> {
// First, try to transfer a request from the requestList
const [listEmpty, listFinished] = await Promise.all([
- this.requestList.isEmpty(),
- this.requestList.isFinished(),
+ this.requestLoader.isEmpty(),
+ this.requestLoader.isFinished(),
]);
if (!listEmpty && !listFinished) {
- await this.transferNextBatchToQueue();
+ // If the transfer failed, the request was dropped; don't fetch from the manager this round (matching
+ // crawlee-python behaviour). The next `fetchNextRequest()` call will pick up where we left off.
+ if (!(await this.transferNextRequestToQueue())) {
+ return null;
+ }
}
- // Try to fetch from queue after potential transfer
- return this.requestQueue.fetchNextRequest<T>();
+ // Try to fetch from manager after the transfer
+ return (await this.getRequestManager()).fetchNextRequest<T>();
}
/**
* @inheritdoc
*/
async isFinished(): Promise<boolean> {
- const storagesFinished = await Promise.all([this.requestList.isFinished(), this.requestQueue.isFinished()]);
+ const requestManager = await this.getRequestManager();
+ const storagesFinished = await Promise.all([this.requestLoader.isFinished(), requestManager.isFinished()]);
return storagesFinished.every(Boolean);
}
@@ -89,30 +122,41 @@ export class RequestManagerTandem implements IRequestManager {
* @inheritdoc
*/
async isEmpty(): Promise<boolean> {
- const storagesEmpty = await Promise.all([this.requestList.isEmpty(), this.requestQueue.isEmpty()]);
+ const requestManager = await this.getRequestManager();
+ const storagesEmpty = await Promise.all([this.requestLoader.isEmpty(), requestManager.isEmpty()]);
return storagesEmpty.every(Boolean);
}
/**
* @inheritdoc
*/
- async handledCount(): Promise<number> {
- // Since one of the stores needs to have priority when both are present, we query the request queue - the request list will first be dumped into the queue and then left empty.
- return await this.requestQueue.handledCount();
+ async getHandledCount(): Promise<number> {
+ // Since one of the stores needs to have priority when both are present, we query the request manager - the request loader will first be dumped into the manager and then left empty.
+ return (await this.getRequestManager()).getHandledCount();
}
/**
* @inheritdoc
*/
- getTotalCount(): number {
- return this.requestQueue.getTotalCount();
+ async getTotalCount(): Promise<number> {
+ const requestManager = await this.getRequestManager();
+ const [managerTotal, loaderTotal] = await Promise.all([
+ requestManager.getTotalCount(),
+ this.requestLoader.getTotalCount(),
+ ]);
+ return managerTotal + loaderTotal;
}
/**
* @inheritdoc
*/
- getPendingCount(): number {
- return this.requestQueue.getPendingCount() + this.requestList.length() - this.requestList.handledCount();
+ async getPendingCount(): Promise<number> {
+ const requestManager = await this.getRequestManager();
+ const [managerPending, loaderPending] = await Promise.all([
+ requestManager.getPendingCount(),
+ this.requestLoader.getPendingCount(),
+ ]);
+ return managerPending + loaderPending;
}
/**
@@ -130,7 +174,7 @@ export class RequestManagerTandem implements IRequestManager {
* @inheritdoc
*/
async markRequestHandled(request: Request): Promise<RequestQueueOperationInfo | void | null> {
- return this.requestQueue.markRequestHandled(request);
+ return (await this.getRequestManager()).markRequestHandled(request);
}
/**
@@ -140,14 +184,14 @@ export class RequestManagerTandem implements IRequestManager {
request: Request,
options?: RequestQueueOperationOptions,
): Promise<RequestQueueOperationInfo | null> {
- return await this.requestQueue.reclaimRequest(request, options);
+ return (await this.getRequestManager()).reclaimRequest(request, options);
}
/**
* @inheritdoc
*/
async addRequest(requestLike: Source, options?: RequestQueueOperationOptions): Promise<RequestQueueOperationInfo> {
- return await this.requestQueue.addRequest(requestLike, options);
+ return (await this.getRequestManager()).addRequest(requestLike, options);
}
/**
@@ -157,6 +201,23 @@ export class RequestManagerTandem implements IRequestManager {
requests: RequestsLike,
options?: AddRequestsBatchedOptions,
): Promise<AddRequestsBatchedResult> {
- return await this.requestQueue.addRequestsBatched(requests, options);
+ return (await this.getRequestManager()).addRequestsBatched(requests, options);
+ }
+
+ /**
+ * Persists the state of the underlying read-only loader, if it supports persistence.
+ * @inheritdoc
+ */
+ async persistState(): Promise<void> {
+ await this.requestLoader.persistState?.();
+ }
+
+ /**
+ * Purges the writable request manager so the tandem can be reused (e.g. across repeated `crawler.run()` calls).
+ * The read-only loader is immutable and cannot be purged, so only the manager side is reset.
+ * @inheritdoc
+ */
+ async purge(): Promise<void> {
+ await (await this.getRequestManager()).purge?.();
}
}
diff --git a/packages/core/src/storages/request_provider.ts b/packages/core/src/storages/request_provider.ts
index abed5d494..99678e370 100644
--- a/packages/core/src/storages/request_provider.ts
+++ b/packages/core/src/storages/request_provider.ts
@@ -34,87 +34,18 @@ import { Request } from '../request.js';
import { serviceLocator } from '../service_locator.js';
import { checkStorageAccess } from './access_checking.js';
import { RequestDeduplicationCache } from './request_dedup_cache.js';
+import type { IRequestManager, RequestsLike } from './request_manager.js';
import type { IStorage, StorageIdentifier } from './storage_instance_manager.js';
import { resolveStorageIdentifier } from './storage_instance_manager.js';
import type { StorageOpenOptions } from './utils.js';
import { getRequestId, purgeDefaultStorages, QUERY_HEAD_MIN_LENGTH } from './utils.js';
-export type RequestsLike = AsyncIterable<Source | string> | Iterable<Source | string> | (Source | string)[];
-
/**
* Maximum number of consecutive batch-add attempts that make no progress before the remaining
* unprocessed requests are skipped, so permanently rejected requests don't retry forever.
*/
const MAX_UNPROCESSED_REQUESTS_RETRIES = 3;
-/**
- * Represents a provider of requests/URLs to crawl.
- */
-export interface IRequestManager {
- /**
- * Returns `true` if all requests were already handled and there are no more left.
- */
- isFinished(): Promise<boolean>;
-
- /**
- * Resolves to `true` if the next call to {@apilink IRequestManager.fetchNextRequest} function
- * would return `null`, otherwise it resolves to `false`.
- * Note that even if the provider is empty, there might be some pending requests currently being processed.
- */
- isEmpty(): Promise<boolean>;
-
- /**
- * Returns number of handled requests.
- */
- handledCount(): Promise<number>;
-
- /**
- * Get the total number of requests known to the request manager.
- */
- getTotalCount(): number;
-
- /**
- * Get an offline approximation of the number of pending requests.
- */
- getPendingCount(): number;
-
- /**
- * Gets the next {@apilink Request} to process.
- *
- * The function's `Promise` resolves to `null` if there are no more
- * requests to process.
- */
- fetchNextRequest<T extends Dictionary = Dictionary>(): Promise<Request<T> | null>;
-
- /**
- * Can be used to iterate over the `RequestManager` instance in a `for await .. of` loop.
- * Provides an alternative for the repeated use of `fetchNextRequest`.
- */
- [Symbol.asyncIterator](): AsyncGenerator<Request>;
-
- /**
- * Marks request as handled after successful processing.
- */
- markRequestHandled(request: Request): Promise<RequestQueueOperationInfo | void | null>;
-
- /**
- * Reclaims request to the provider if its processing failed.
- * The request will become available in the next `fetchNextRequest()`.
- */
- reclaimRequest(request: Request, options?: RequestQueueOperationOptions): Promise<RequestQueueOperationInfo | null>;
-
- addRequest(requestLike: Source, options?: RequestQueueOperationOptions): Promise<RequestQueueOperationInfo>;
-
- addRequestsBatched(requests: RequestsLike, options?: AddRequestsBatchedOptions): Promise<AddRequestsBatchedResult>;
-
- /**
- * Remove all requests from the queue but keep the queue itself, resetting it
- * so it can be reused (e.g. across multiple `crawler.run()` calls).
- *
- * Implementations that do not support purging may leave this `undefined`.
- */
- purge?(): Promise<void>;
-}
export abstract class RequestProvider implements IStorage, IRequestManager {
id: string;
@@ -133,8 +64,6 @@ export abstract class RequestProvider implements IStorage, IRequestManager {
assumedTotalCount = 0;
assumedHandledCount = 0;
- private initialCount = 0;
- private initialHandledCount = 0; // We track this separately from `assumedHandledCount` which is used non-trivially by RequestQueueV1
private isInitialized = false;
protected queueHeadIds = new ListDictionary<string>();
@@ -185,21 +114,23 @@ export abstract class RequestProvider implements IStorage, IRequestManager {
}
/**
- * Returns an offline approximation of the total number of requests in the queue (i.e. pending + handled).
+ * Returns the total number of requests in the queue (i.e. pending + handled).
*
* Survives restarts and actor migrations.
*/
- getTotalCount() {
- return this.assumedTotalCount + this.initialCount;
+ async getTotalCount() {
+ const { totalRequestCount } = await this.getInfo();
+ return totalRequestCount;
}
/**
- * Returns an offline approximation of the total number of pending requests in the queue.
+ * Returns the total number of pending requests in the queue.
*
* Survives restarts and Actor migrations.
*/
- getPendingCount() {
- return this.getTotalCount() - this.initialHandledCount - this.assumedHandledCount;
+ async getPendingCount() {
+ const { totalRequestCount, handledRequestCount } = await this.getInfo();
+ return totalRequestCount - handledRequestCount;
}
/**
@@ -828,8 +759,6 @@ export abstract class RequestProvider implements IStorage, IRequestManager {
// Reset in-memory bookkeeping so the queue behaves as if freshly opened.
this.assumedTotalCount = 0;
this.assumedHandledCount = 0;
- this.initialCount = 0;
- this.initialHandledCount = 0;
this.queueHeadIds.clear();
this.requestCache.clear();
this.recentlyHandledRequestsCache.clear();
@@ -859,8 +788,8 @@ export abstract class RequestProvider implements IStorage, IRequestManager {
* ```
* @inheritdoc
*/
- async handledCount(): Promise<number> {
- // NOTE: We keep this function for compatibility with RequestList.handledCount()
+ async getHandledCount(): Promise<number> {
+ // NOTE: We keep this function for compatibility with RequestList.getHandledCount()
const { handledRequestCount } = await this.getInfo();
return handledRequestCount;
}
@@ -1016,10 +945,6 @@ export abstract class RequestProvider implements IStorage, IRequestManager {
timeoutSecs: queue.timeoutSecs,
});
- const queueInfo = await queue.client.getMetadata();
-
- queue.initialCount = queueInfo.totalRequestCount;
- queue.initialHandledCount = queueInfo.handledRequestCount;
queue.isInitialized = true;
}
diff --git a/packages/core/src/storages/request_queue_v2.ts b/packages/core/src/storages/request_queue_v2.ts
index dcacd16bc..a37a9587c 100644
--- a/packages/core/src/storages/request_queue_v2.ts
+++ b/packages/core/src/storages/request_queue_v2.ts
@@ -5,11 +5,11 @@ import { EventType } from '../events/event_manager.js';
import type { Request, Source } from '../request.js';
import { serviceLocator } from '../service_locator.js';
import { checkStorageAccess } from './access_checking.js';
+import type { RequestsLike } from './request_manager.js';
import type {
RequestProviderOptions,
RequestQueueOperationInfo,
RequestQueueOperationOptions,
- RequestsLike,
} from './request_provider.js';
import { RequestProvider } from './request_provider.js';
import { getRequestId } from './utils.js';
diff --git a/packages/core/src/storages/sitemap_request_list.ts b/packages/core/src/storages/sitemap_request_loader.ts
similarity index 87%
rename from packages/core/src/storages/sitemap_request_list.ts
rename to packages/core/src/storages/sitemap_request_loader.ts
index 0105b5074..0ad8b9cb2 100644
--- a/packages/core/src/storages/sitemap_request_list.ts
+++ b/packages/core/src/storages/sitemap_request_loader.ts
@@ -13,11 +13,12 @@ import type { CrawleeLogger } from '../log.js';
import { Request } from '../request.js';
import { serviceLocator } from '../service_locator.js';
import { KeyValueStore } from './key_value_store.js';
-import type { IRequestList } from './request_list.js';
+import type { IRequestLoader } from './request_loader.js';
+import type { IRequestManager } from './request_manager.js';
import { purgeDefaultStorages } from './utils.js';
/** @internal */
-const STATE_PERSISTENCE_KEY = 'SITEMAP_REQUEST_LIST_STATE';
+const STATE_PERSISTENCE_KEY = 'SITEMAP_REQUEST_LOADER_STATE';
interface UrlConstraints {
/**
@@ -29,7 +30,7 @@ interface UrlConstraints {
* The matching is always case-insensitive.
* If you need case-sensitive matching, use `regexps` property directly.
*
- * If `globs` is an empty array or `undefined`, and `regexps` are also not defined, then the `SitemapRequestList`
+ * If `globs` is an empty array or `undefined`, and `regexps` are also not defined, then the `SitemapRequestLoader`
* includes all the URLs from the sitemap.
*/
globs?: readonly GlobInput[];
@@ -51,13 +52,13 @@ interface UrlConstraints {
*
* The plain objects must include at least the `regexp` property, which holds the regular expression.
*
- * If `regexps` is an empty array or `undefined`, and `globs` are also not defined, then the `SitemapRequestList`
+ * If `regexps` is an empty array or `undefined`, and `globs` are also not defined, then the `SitemapRequestLoader`
* includes all the URLs from the sitemap.
*/
regexps?: readonly RegExpInput[];
}
-export interface SitemapRequestListOptions extends UrlConstraints {
+export interface SitemapRequestLoaderOptions extends UrlConstraints {
/**
* List of sitemap URLs to parse.
*/
@@ -118,9 +119,8 @@ interface SitemapParsingProgress {
pendingSitemapUrls: Set<string>;
}
-interface SitemapRequestListState {
+interface SitemapRequestLoaderState {
urlQueue: string[];
- reclaimed: string[];
sitemapParsingProgress: Record<keyof SitemapParsingProgress, any>;
abortLoading: boolean;
closed: boolean;
@@ -132,20 +132,17 @@ interface SitemapRequestListState {
*
* The loading of the sitemap is performed in the background so that crawling can start before the sitemap is fully loaded.
*/
-export class SitemapRequestList implements IRequestList {
+export class SitemapRequestLoader implements IRequestLoader {
/**
* Set of URLs that were returned by `fetchNextRequest()` and not marked as handled yet.
* @internal
*/
inProgress = new Set<string>();
- /** Set of URLs for which `reclaimRequest()` was called. */
- private reclaimed = new Set<string>();
-
/**
* Map of returned Request objects that have not been marked as handled yet.
*
- * We use this to persist custom user fields on the in-progress (or reclaimed) requests.
+ * We use this to persist custom user fields on the in-progress requests.
*/
private requestData = new Map<string, Request>();
@@ -215,10 +212,10 @@ export class SitemapRequestList implements IRequestList {
/** EventManager used to handle persistence */
private events: EventManager;
- private persistenceOptions: RequiredDeep<SitemapRequestListOptions['persistenceOptions']>;
+ private persistenceOptions: RequiredDeep<SitemapRequestLoaderOptions['persistenceOptions']>;
/** @internal */
- private constructor(options: SitemapRequestListOptions) {
+ private constructor(options: SitemapRequestLoaderOptions) {
ow(
options,
ow.object.exactShape({
@@ -242,7 +239,7 @@ export class SitemapRequestList implements IRequestList {
const { globs, exclude, regexps } = options;
- this.log = serviceLocator.getLogger().child({ prefix: 'SitemapRequestList' });
+ this.log = serviceLocator.getLogger().child({ prefix: 'SitemapRequestLoader' });
if (exclude?.length) {
for (const excl of exclude) {
@@ -386,7 +383,7 @@ export class SitemapRequestList implements IRequestList {
private async load({
parseSitemapOptions,
}: {
- parseSitemapOptions?: SitemapRequestListOptions['parseSitemapOptions'];
+ parseSitemapOptions?: SitemapRequestLoaderOptions['parseSitemapOptions'];
}): Promise<void> {
while (!this.isSitemapFullyLoaded() && !this.abortLoading) {
const sitemapUrl =
@@ -426,14 +423,14 @@ export class SitemapRequestList implements IRequestList {
/**
* Open a sitemap and start processing it.
*
- * Resolves to a new instance of `SitemapRequestList`, which **might not be fully loaded yet** - i.e. the sitemap might still be loading in the background.
+ * Resolves to a new instance of `SitemapRequestLoader`, which **might not be fully loaded yet** - i.e. the sitemap might still be loading in the background.
*
* Track the loading progress using the `isSitemapFullyLoaded` property.
*/
- static async open(options: SitemapRequestListOptions): Promise<SitemapRequestList> {
+ static async open(options: SitemapRequestLoaderOptions): Promise<SitemapRequestLoader> {
const { httpClient, ...restOptions } = options;
- const requestList = new SitemapRequestList({
+ const requestList = new SitemapRequestLoader({
...restOptions,
persistStateKey: options.persistStateKey ?? STATE_PERSISTENCE_KEY,
});
@@ -462,8 +459,31 @@ export class SitemapRequestList implements IRequestList {
/**
* @inheritDoc
*/
- length(): number {
- return this.urlQueueStream.readableLength + this.handledUrlCount - this.inProgress.size - this.reclaimed.size;
+ async getTotalCount(): Promise<number> {
+ // Total known so far = not-yet-fetched (still buffered in the stream) + in-progress (fetched but not
+ // yet handled) + already handled.
+ return this.urlQueueStream.readableLength + this.inProgress.size + this.handledUrlCount;
+ }
+
+ /**
+ * @inheritDoc
+ */
+ async getPendingCount(): Promise<number> {
+ // Pending = everything not yet handled = not-yet-fetched + in-progress.
+ return this.urlQueueStream.readableLength + this.inProgress.size;
+ }
+
+ /**
+ * Combines this list with a request manager (a {@apilink RequestQueue} by default) into a
+ * {@apilink RequestManagerTandem}, allowing requests to be added and reclaimed while still
+ * being read from this list first.
+ */
+ async toTandem(requestManager?: IRequestManager): Promise<IRequestManager> {
+ // Import here to avoid circular imports.
+ const { RequestManagerTandem } = await import('./request_manager_tandem.js');
+ const { RequestQueue } = await import('./request_queue_v2.js');
+
+ return new RequestManagerTandem(this, requestManager ?? (await RequestQueue.open()));
}
/**
@@ -479,13 +499,13 @@ export class SitemapRequestList implements IRequestList {
* @inheritDoc
*/
async isEmpty(): Promise<boolean> {
- return this.reclaimed.size === 0 && this.urlQueueStream.readableLength === 0;
+ return this.urlQueueStream.readableLength === 0;
}
/**
* @inheritDoc
*/
- handledCount(): number {
+ async getHandledCount(): Promise<number> {
return this.handledUrlCount;
}
@@ -537,12 +557,12 @@ export class SitemapRequestList implements IRequestList {
inProgressSitemapUrl: this.sitemapParsingProgress.inProgressSitemapUrl,
inProgressEntries: Array.from(this.sitemapParsingProgress.inProgressEntries),
},
- urlQueue,
- reclaimed: [...this.inProgress, ...this.reclaimed], // In-progress and reclaimed requests will be both retried if state is restored
+ // Re-queue in-progress requests to the front so they are retried if the state is restored.
+ urlQueue: [...this.inProgress, ...urlQueue],
requestData: Array.from(this.requestData.entries()),
abortLoading: this.abortLoading,
closed: this.closed,
- } satisfies SitemapRequestListState);
+ } satisfies SitemapRequestLoaderState);
}
private async restoreState(): Promise<void> {
@@ -553,13 +573,12 @@ export class SitemapRequestList implements IRequestList {
}
this.store ??= await KeyValueStore.open();
- const state = await this.store.getValue<SitemapRequestListState>(this.persistStateKey);
+ const state = await this.store.getValue<SitemapRequestLoaderState>(this.persistStateKey);
if (state === null) {
return;
}
- this.reclaimed = new Set(state.reclaimed);
this.sitemapParsingProgress = {
pendingSitemapUrls: new Set(state.sitemapParsingProgress.pendingSitemapUrls),
inProgressSitemapUrl: state.sitemapParsingProgress.inProgressSitemapUrl,
@@ -580,16 +599,13 @@ export class SitemapRequestList implements IRequestList {
* @inheritDoc
*/
async fetchNextRequest(): Promise<Request | null> {
- // Try to return a reclaimed request first
- let nextUrl: string | undefined | null = this.reclaimed.values().next().value;
- if (nextUrl) {
- this.reclaimed.delete(nextUrl);
- } else {
- // Otherwise read next url from the stream
- nextUrl = await this.readNextUrl();
- if (!nextUrl) {
- return null;
- }
+ const nextUrl = await this.readNextUrl();
+ if (!nextUrl) {
+ return null;
+ }
+
+ // A restored in-progress request already has its Request data; don't overwrite it.
+ if (!this.requestData.has(nextUrl)) {
this.requestData.set(nextUrl, new Request({ url: nextUrl, enqueueStrategy: this.enqueueStrategy }));
}
@@ -609,15 +625,6 @@ export class SitemapRequestList implements IRequestList {
}
}
- /**
- * @inheritDoc
- */
- async reclaimRequest(request: Request): Promise<void> {
- this.ensureInProgressAndNotReclaimed(request.url);
- this.reclaimed.add(request.url);
- this.inProgress.delete(request.url);
- }
-
/**
* Aborts the internal sitemap loading, stops the processing of the sitemap contents and drops all the pending URLs.
*
@@ -637,17 +644,14 @@ export class SitemapRequestList implements IRequestList {
*/
async markRequestHandled(request: Request): Promise<void> {
this.handledUrlCount += 1;
- this.ensureInProgressAndNotReclaimed(request.url);
+ this.ensureInProgress(request.url);
this.inProgress.delete(request.url);
this.requestData.delete(request.url);
}
- private ensureInProgressAndNotReclaimed(url: string): void {
+ private ensureInProgress(url: string): void {
if (!this.inProgress.has(url)) {
throw new Error(`The request is not being processed (url: ${url})`);
}
- if (this.reclaimed.has(url)) {
- throw new Error(`The request was already reclaimed (url: ${url})`);
- }
}
}
diff --git a/packages/core/test/enqueue_links/protocol-matching-based-on-strategy.test.ts b/packages/core/test/enqueue_links/protocol-matching-based-on-strategy.test.ts
index 576f86587..70c379d95 100644
--- a/packages/core/test/enqueue_links/protocol-matching-based-on-strategy.test.ts
+++ b/packages/core/test/enqueue_links/protocol-matching-based-on-strategy.test.ts
@@ -55,7 +55,7 @@ describe('enqueueLinks() - matching and ignoring http/https protocol differences
await cheerioCrawlerEnqueueLinks({
options: { selector: 'a', strategy: EnqueueStrategy.SameHostname },
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -70,7 +70,7 @@ describe('enqueueLinks() - matching and ignoring http/https protocol differences
await cheerioCrawlerEnqueueLinks({
options: { selector: 'a', strategy: EnqueueStrategy.SameDomain },
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'http://example.com',
});
@@ -85,7 +85,7 @@ describe('enqueueLinks() - matching and ignoring http/https protocol differences
await cheerioCrawlerEnqueueLinks({
options: { selector: 'a', strategy: EnqueueStrategy.SameOrigin },
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
diff --git a/packages/core/test/enqueue_links/user-provided-patterns-with-enqueue-strategy.test.ts b/packages/core/test/enqueue_links/user-provided-patterns-with-enqueue-strategy.test.ts
index bac01890f..468fedffe 100644
--- a/packages/core/test/enqueue_links/user-provided-patterns-with-enqueue-strategy.test.ts
+++ b/packages/core/test/enqueue_links/user-provided-patterns-with-enqueue-strategy.test.ts
@@ -73,7 +73,7 @@ describe('enqueueLinks() - combining user patterns with enqueue strategies', ()
strategy: EnqueueStrategy.SameDomain,
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -94,7 +94,7 @@ describe('enqueueLinks() - combining user patterns with enqueue strategies', ()
strategy: EnqueueStrategy.All,
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -113,7 +113,7 @@ describe('enqueueLinks() - combining user patterns with enqueue strategies', ()
strategy: EnqueueStrategy.SameDomain,
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -135,7 +135,7 @@ describe('enqueueLinks() - combining user patterns with enqueue strategies', ()
exclude,
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -154,7 +154,7 @@ describe('enqueueLinks() - combining user patterns with enqueue strategies', ()
strategy: EnqueueStrategy.All,
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
diff --git a/packages/core/test/enqueue_links/userData.test.ts b/packages/core/test/enqueue_links/userData.test.ts
index 74f3cf93f..282d3910a 100644
--- a/packages/core/test/enqueue_links/userData.test.ts
+++ b/packages/core/test/enqueue_links/userData.test.ts
@@ -61,7 +61,7 @@ describe("enqueueLinks() - userData shouldn't be changed and outer label must ta
label: 'first',
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -72,7 +72,7 @@ describe("enqueueLinks() - userData shouldn't be changed and outer label must ta
label: 'second',
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -96,7 +96,7 @@ describe("enqueueLinks() - userData shouldn't be changed and outer label must ta
label: 'first',
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
const userDataAfterEnqueue = JSON.stringify(userData);
@@ -114,7 +114,7 @@ describe("enqueueLinks() - userData shouldn't be changed and outer label must ta
sessionId: 'my-session',
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -129,7 +129,7 @@ describe("enqueueLinks() - userData shouldn't be changed and outer label must ta
await cheerioCrawlerEnqueueLinks({
options: {},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
diff --git a/packages/http-crawler/src/internals/file-download.ts b/packages/http-crawler/src/internals/file-download.ts
index fbb09d994..81dd217d0 100644
--- a/packages/http-crawler/src/internals/file-download.ts
+++ b/packages/http-crawler/src/internals/file-download.ts
@@ -123,9 +123,9 @@ export function ByteCounterStream({
*
* `FileCrawler` downloads each URL using a plain HTTP request and then invokes the user-provided {@apilink FileDownloadOptions.requestHandler} where the user can specify what to do with the downloaded data.
*
- * The source URLs are represented using {@apilink Request} objects that are fed from {@apilink RequestList} or {@apilink RequestQueue} instances provided by the {@apilink FileDownloadOptions.requestList} or {@apilink FileDownloadOptions.requestQueue} constructor options, respectively.
+ * The source URLs are represented using {@apilink Request} objects that are fed from the {@apilink IRequestManager|request manager} provided via the {@apilink FileDownloadOptions.requestManager|`requestManager`} constructor option (a {@apilink RequestQueue} is itself a request manager). To read from a read-only source such as a {@apilink RequestList} while still being able to enqueue new requests, combine it with a queue into a {@apilink RequestManagerTandem} via {@apilink IRequestLoader.toTandem|`requestLoader.toTandem()`} and pass the result as `requestManager`.
*
- * If both {@apilink FileDownloadOptions.requestList} and {@apilink FileDownloadOptions.requestQueue} are used, the instance first processes URLs from the {@apilink RequestList} and automatically enqueues all of them to {@apilink RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
+ * > The {@apilink FileDownloadOptions.requestList|`requestList`} and {@apilink FileDownloadOptions.requestQueue|`requestQueue`} options are deprecated; they are still accepted and folded into a single `requestManager` for back-compat.
*
* The crawler finishes when there are no more {@apilink Request} objects to crawl.
*
diff --git a/packages/http-crawler/src/internals/http-crawler.ts b/packages/http-crawler/src/internals/http-crawler.ts
index 37e6b54a6..bea23a631 100644
--- a/packages/http-crawler/src/internals/http-crawler.ts
+++ b/packages/http-crawler/src/internals/http-crawler.ts
@@ -253,13 +253,15 @@ export type HttpRequestHandler<
*
* This crawler downloads each URL using a plain HTTP request and doesn't do any HTML parsing.
*
- * The source URLs are represented using {@apilink Request} objects that are fed from
- * {@apilink RequestList} or {@apilink RequestQueue} instances provided by the {@apilink HttpCrawlerOptions.requestList}
- * or {@apilink HttpCrawlerOptions.requestQueue} constructor options, respectively.
+ * The source URLs are represented using {@apilink Request} objects that are fed from the
+ * {@apilink IRequestManager|request manager} provided via the {@apilink HttpCrawlerOptions.requestManager|`requestManager`}
+ * constructor option (a {@apilink RequestQueue} is itself a request manager). To read from a read-only source such
+ * as a {@apilink RequestList} while still being able to enqueue new requests, combine it with a queue into a
+ * {@apilink RequestManagerTandem} via {@apilink IRequestLoader.toTandem|`requestLoader.toTandem()`} and pass the
+ * result as `requestManager`.
*
- * If both {@apilink HttpCrawlerOptions.requestList} and {@apilink HttpCrawlerOptions.requestQueue} are used,
- * the instance first processes URLs from the {@apilink RequestList} and automatically enqueues all of them
- * to {@apilink RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
+ * > The {@apilink HttpCrawlerOptions.requestList|`requestList`} and {@apilink HttpCrawlerOptions.requestQueue|`requestQueue`}
+ * > options are deprecated; they are still accepted and folded into a single `requestManager` for back-compat.
*
* The crawler finishes when there are no more {@apilink Request} objects to crawl.
*
diff --git a/packages/jsdom-crawler/src/internals/jsdom-crawler.ts b/packages/jsdom-crawler/src/internals/jsdom-crawler.ts
index 7323af64a..7d35f59c6 100644
--- a/packages/jsdom-crawler/src/internals/jsdom-crawler.ts
+++ b/packages/jsdom-crawler/src/internals/jsdom-crawler.ts
@@ -6,8 +6,8 @@ import type {
HttpCrawlerOptions,
InternalHttpCrawlingContext,
InternalHttpHook,
+ IRequestManager,
RequestHandler,
- RequestProvider,
RouterHandler,
RouterRoutes,
RouteSchemas,
@@ -123,13 +123,15 @@ export type JSDOMRequestHandler<
* and then invokes the user-provided {@apilink JSDOMCrawlerOptions.requestHandler} to extract page data
* using the `window` object.
*
- * The source URLs are represented using {@apilink Request} objects that are fed from
- * {@apilink RequestList} or {@apilink RequestQueue} instances provided by the {@apilink JSDOMCrawlerOptions.requestList}
- * or {@apilink JSDOMCrawlerOptions.requestQueue} constructor options, respectively.
+ * The source URLs are represented using {@apilink Request} objects that are fed from the
+ * {@apilink IRequestManager|request manager} provided via the {@apilink JSDOMCrawlerOptions.requestManager|`requestManager`}
+ * constructor option (a {@apilink RequestQueue} is itself a request manager). To read from a read-only source such
+ * as a {@apilink RequestList} while still being able to enqueue new requests, combine it with a queue into a
+ * {@apilink RequestManagerTandem} via {@apilink IRequestLoader.toTandem|`requestLoader.toTandem()`} and pass the
+ * result as `requestManager`.
*
- * If both {@apilink JSDOMCrawlerOptions.requestList} and {@apilink JSDOMCrawlerOptions.requestQueue} are used,
- * the instance first processes URLs from the {@apilink RequestList} and automatically enqueues all of them
- * to {@apilink RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
+ * > The {@apilink JSDOMCrawlerOptions.requestList|`requestList`} and {@apilink JSDOMCrawlerOptions.requestQueue|`requestQueue`}
+ * > options are deprecated; they are still accepted and folded into a single `requestManager` for back-compat.
*
* The crawler finishes when there are no more {@apilink Request} objects to crawl.
*
@@ -356,7 +358,7 @@ export class JSDOMCrawler<
// pre-clamping it here would make the crawler log the internal limit as a user-provided one
options: enqueueOptions,
window: crawlingContext.window,
- requestQueue: await this.getRequestQueue(),
+ requestManager: await this.getRequestManager(),
robotsTxtFile: await this.getRobotsTxtFileForUrl(crawlingContext.request.url),
onSkippedRequest: this.handleSkippedRequest,
originalRequestUrl: crawlingContext.request.url,
@@ -393,7 +395,7 @@ export class JSDOMCrawler<
interface EnqueueLinksInternalOptions {
options?: EnqueueLinksOptions;
window: DOMWindow | null;
- requestQueue: RequestProvider;
+ requestManager: IRequestManager;
robotsTxtFile?: RobotsTxtFile;
onSkippedRequest?: SkippedRequestCallback;
originalRequestUrl: string;
@@ -445,7 +447,7 @@ export async function domCrawlerEnqueueLinks(options: EnqueueLinksInternalOption
}
return enqueueLinks({
- requestQueue: options.requestQueue,
+ requestManager: options.requestManager,
robotsTxtFile: options.robotsTxtFile,
onSkippedRequest: options.onSkippedRequest,
urls,
diff --git a/packages/linkedom-crawler/src/internals/linkedom-crawler.ts b/packages/linkedom-crawler/src/internals/linkedom-crawler.ts
index 77ead2cd7..7e0b76842 100644
--- a/packages/linkedom-crawler/src/internals/linkedom-crawler.ts
+++ b/packages/linkedom-crawler/src/internals/linkedom-crawler.ts
@@ -6,8 +6,8 @@ import type {
HttpCrawlerOptions,
InternalHttpCrawlingContext,
InternalHttpHook,
+ IRequestManager,
RequestHandler,
- RequestProvider,
RouterHandler,
RouterRoutes,
RouteSchemas,
@@ -39,7 +39,7 @@ export interface LinkeDOMCrawlerOptions<
JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
> extends HttpCrawlerOptions<LinkeDOMCrawlingContext<UserData, JSONData>, ContextExtension, ExtendedContext> {}
-export interface LinkeDOMCrawlerEnqueueLinksOptions extends Omit<EnqueueLinksOptions, 'urls' | 'requestQueue'> {}
+export interface LinkeDOMCrawlerEnqueueLinksOptions extends Omit<EnqueueLinksOptions, 'urls' | 'requestManager'> {}
export type LinkeDOMHook<
UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
@@ -112,13 +112,15 @@ export type LinkeDOMRequestHandler<
* and then invokes the user-provided {@apilink LinkeDOMCrawlerOptions.requestHandler} to extract page data
* using the `window` object.
*
- * The source URLs are represented using {@apilink Request} objects that are fed from
- * {@apilink RequestList} or {@apilink RequestQueue} instances provided by the {@apilink LinkeDOMCrawlerOptions.requestList}
- * or {@apilink LinkeDOMCrawlerOptions.requestQueue} constructor options, respectively.
+ * The source URLs are represented using {@apilink Request} objects that are fed from the
+ * {@apilink IRequestManager|request manager} provided via the {@apilink LinkeDOMCrawlerOptions.requestManager|`requestManager`}
+ * constructor option (a {@apilink RequestQueue} is itself a request manager). To read from a read-only source such
+ * as a {@apilink RequestList} while still being able to enqueue new requests, combine it with a queue into a
+ * {@apilink RequestManagerTandem} via {@apilink IRequestLoader.toTandem|`requestLoader.toTandem()`} and pass the
+ * result as `requestManager`.
*
- * If both {@apilink LinkeDOMCrawlerOptions.requestList} and {@apilink LinkeDOMCrawlerOptions.requestQueue} are used,
- * the instance first processes URLs from the {@apilink RequestList} and automatically enqueues all of them
- * to {@apilink RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
+ * > The {@apilink LinkeDOMCrawlerOptions.requestList|`requestList`} and {@apilink LinkeDOMCrawlerOptions.requestQueue|`requestQueue`}
+ * > options are deprecated; they are still accepted and folded into a single `requestManager` for back-compat.
*
* The crawler finishes when there are no more {@apilink Request} objects to crawl.
*
@@ -244,7 +246,7 @@ export class LinkeDOMCrawler<
// pre-clamping it here would make the crawler log the internal limit as a user-provided one
options: enqueueOptions,
window: document.defaultView,
- requestQueue: await this.getRequestQueue(),
+ requestManager: await this.getRequestManager(),
robotsTxtFile: await this.getRobotsTxtFileForUrl(crawlingContext.request.url),
onSkippedRequest: this.handleSkippedRequest,
originalRequestUrl: crawlingContext.request.url,
@@ -281,7 +283,7 @@ export class LinkeDOMCrawler<
interface EnqueueLinksInternalOptions {
options?: EnqueueLinksOptions;
window: Window | null;
- requestQueue: RequestProvider;
+ requestManager: IRequestManager;
robotsTxtFile?: RobotsTxtFile;
onSkippedRequest?: SkippedRequestCallback;
originalRequestUrl: string;
@@ -335,7 +337,7 @@ export async function linkedomCrawlerEnqueueLinks(
}
return enqueueLinks({
- requestQueue: options.requestQueue,
+ requestManager: options.requestManager,
robotsTxtFile: options.robotsTxtFile,
onSkippedRequest: options.onSkippedRequest,
urls,
diff --git a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts
index 09910967f..d31fe77db 100644
--- a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts
+++ b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts
@@ -755,6 +755,15 @@ export class AdaptivePlaywrightCrawler<
);
}
+ /**
+ * Reading the pending request count queries the underlying request manager, which counts as storage access.
+ * Since this is internal crawler bookkeeping used to compute the `enqueueLinks` limit (not user-initiated storage
+ * access), it must be allowed even while a request handler runs inside the storage-access guard.
+ */
+ protected override async getPendingRequestCountApproximation(): Promise<number> {
+ return this.allowStorageAccess(() => super.getPendingRequestCountApproximation())();
+ }
+
protected async enqueueLinks(
options: SetRequired<EnqueueLinksOptions, 'urls'>,
request: RestrictedCrawlingContext['request'],
diff --git a/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts b/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts
index 228bab85a..3d56271a1 100644
--- a/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts
+++ b/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts
@@ -2,10 +2,10 @@ import { URL } from 'node:url';
import type {
GlobInput,
+ IRequestManager,
PseudoUrlInput,
RegExpInput,
RequestOptions,
- RequestProvider,
RequestTransform,
SkippedRequestCallback,
UrlPatternObject,
@@ -36,9 +36,9 @@ export interface EnqueueLinksByClickingElementsOptions {
page: Page;
/**
- * A request queue to which the URLs will be enqueued.
+ * * A request manager to which the URLs will be enqueued.
*/
- requestQueue: RequestProvider;
+ requestManager: IRequestManager;
/**
* A CSS selector matching elements to be clicked on. Unlike in {@apilink enqueueLinks}, there is no default
@@ -223,7 +223,7 @@ export interface EnqueueLinksByClickingElementsOptions {
* ```javascript
* await playwrightUtils.enqueueLinksByClickingElements({
* page,
- * requestQueue,
+ * requestManager,
* selector: 'a.product-detail',
* pseudoUrls: [
* 'https://www.example.com/handbags/[.*]'
@@ -241,7 +241,7 @@ export async function enqueueLinksByClickingElements(
options,
ow.object.exactShape({
page: ow.object.hasKeys('goto', 'evaluate'),
- requestQueue: ow.object.hasKeys('fetchNextRequest', 'addRequest'),
+ requestManager: ow.object.hasKeys('fetchNextRequest', 'addRequestsBatched'),
selector: ow.string,
userData: ow.optional.object,
clickOptions: ow.optional.object,
@@ -263,7 +263,7 @@ export async function enqueueLinksByClickingElements(
const {
page,
- requestQueue,
+ requestManager,
selector,
clickOptions,
// oxlint-disable-next-line typescript/no-deprecated -- still accepted for backwards compat
@@ -341,7 +341,7 @@ export async function enqueueLinksByClickingElements(
}
const requests = filteredOptions.map((opts) => new CrawleeRequest(opts));
- const { addedRequests } = await requestQueue.addRequestsBatched(requests, { forefront });
+ const { addedRequests } = await requestManager.addRequestsBatched(requests, { forefront });
return { processedRequests: addedRequests, unprocessedRequests: [] };
}
diff --git a/packages/playwright-crawler/src/internals/playwright-crawler.ts b/packages/playwright-crawler/src/internals/playwright-crawler.ts
index 8633ba6a5..f900adfd8 100644
--- a/packages/playwright-crawler/src/internals/playwright-crawler.ts
+++ b/packages/playwright-crawler/src/internals/playwright-crawler.ts
@@ -128,13 +128,15 @@ export interface PlaywrightCrawlerOptions<
* If the target website doesn't need JavaScript, consider using {@apilink CheerioCrawler},
* which downloads the pages using raw HTTP requests and is about 10x faster.
*
- * The source URLs are represented using {@apilink Request} objects that are fed from
- * {@apilink RequestList} or {@apilink RequestQueue} instances provided by the {@apilink PlaywrightCrawlerOptions.requestList}
- * or {@apilink PlaywrightCrawlerOptions.requestQueue} constructor options, respectively.
+ * The source URLs are represented using {@apilink Request} objects that are fed from the
+ * {@apilink IRequestManager|request manager} provided via the {@apilink PlaywrightCrawlerOptions.requestManager|`requestManager`}
+ * constructor option (a {@apilink RequestQueue} is itself a request manager). To read from a read-only source such
+ * as a {@apilink RequestList} while still being able to enqueue new requests, combine it with a queue into a
+ * {@apilink RequestManagerTandem} via {@apilink IRequestLoader.toTandem|`requestLoader.toTandem()`} and pass the
+ * result as `requestManager`.
*
- * If both {@apilink PlaywrightCrawlerOptions.requestList} and {@apilink PlaywrightCrawlerOptions.requestQueue} are used,
- * the instance first processes URLs from the {@apilink RequestList} and automatically enqueues all of them
- * to {@apilink RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
+ * > The {@apilink PlaywrightCrawlerOptions.requestList|`requestList`} and {@apilink PlaywrightCrawlerOptions.requestQueue|`requestQueue`}
+ * > options are deprecated; they are still accepted and folded into a single `requestManager` for back-compat.
*
* The crawler finishes when there are no more {@apilink Request} objects to crawl.
*
@@ -292,12 +294,12 @@ export class PlaywrightCrawler<
saveSnapshot: async (options?: SaveSnapshotOptions) =>
playwrightUtils.saveSnapshot(context.page, { ...options, config: serviceLocator.getConfiguration() }),
enqueueLinksByClickingElements: async (
- options: Omit<EnqueueLinksByClickingElementsOptions, 'page' | 'requestQueue'>,
+ options: Omit<EnqueueLinksByClickingElementsOptions, 'page' | 'requestManager'>,
) =>
playwrightUtils.enqueueLinksByClickingElements({
...options,
page: context.page,
- requestQueue: this.requestQueue!,
+ requestManager: this.requestManager!,
}),
compileScript: (scriptString: string, ctx?: Dictionary) => playwrightUtils.compileScript(scriptString, ctx),
closeCookieModals: async () => playwrightUtils.closeCookieModals(context.page),
diff --git a/packages/playwright-crawler/src/internals/utils/playwright-utils.ts b/packages/playwright-crawler/src/internals/utils/playwright-utils.ts
index 11b634eee..00e83d401 100644
--- a/packages/playwright-crawler/src/internals/utils/playwright-utils.ts
+++ b/packages/playwright-crawler/src/internals/utils/playwright-utils.ts
@@ -997,7 +997,7 @@ export interface PlaywrightContextUtils {
* @returns Promise that resolves to {@apilink BatchAddRequestsResult} object.
*/
enqueueLinksByClickingElements(
- options: Omit<EnqueueLinksByClickingElementsOptions, 'page' | 'requestQueue'>,
+ options: Omit<EnqueueLinksByClickingElementsOptions, 'page' | 'requestManager'>,
): Promise<BatchAddRequestsResult>;
/**
diff --git a/packages/puppeteer-crawler/src/internals/enqueue-links/click-elements.ts b/packages/puppeteer-crawler/src/internals/enqueue-links/click-elements.ts
index 5ac3d49ee..17babad6c 100644
--- a/packages/puppeteer-crawler/src/internals/enqueue-links/click-elements.ts
+++ b/packages/puppeteer-crawler/src/internals/enqueue-links/click-elements.ts
@@ -2,10 +2,10 @@ import { URL } from 'node:url';
import type {
GlobInput,
+ IRequestManager,
PseudoUrlInput,
RegExpInput,
RequestOptions,
- RequestProvider,
RequestTransform,
SkippedRequestCallback,
UrlPatternObject,
@@ -37,9 +37,9 @@ export interface EnqueueLinksByClickingElementsOptions {
page: Page;
/**
- * A request queue to which the URLs will be enqueued.
+ * * A request manager to which the URLs will be enqueued.
*/
- requestQueue: RequestProvider;
+ requestManager: IRequestManager;
/**
* A CSS selector matching elements to be clicked on. Unlike in {@apilink enqueueLinks}, there is no default
@@ -224,7 +224,7 @@ export interface EnqueueLinksByClickingElementsOptions {
* ```javascript
* await utils.puppeteer.enqueueLinksByClickingElements({
* page,
- * requestQueue,
+ * requestManager,
* selector: 'a.product-detail',
* pseudoUrls: [
* 'https://www.example.com/handbags/[.*]'
@@ -242,7 +242,7 @@ export async function enqueueLinksByClickingElements(
options,
ow.object.exactShape({
page: ow.object.hasKeys('goto', 'evaluate'),
- requestQueue: ow.object.hasKeys('fetchNextRequest', 'addRequest'),
+ requestManager: ow.object.hasKeys('fetchNextRequest', 'addRequestsBatched'),
selector: ow.string,
userData: ow.optional.object,
clickOptions: ow.optional.object,
@@ -264,7 +264,7 @@ export async function enqueueLinksByClickingElements(
const {
page,
- requestQueue,
+ requestManager,
selector,
clickOptions,
// oxlint-disable-next-line typescript/no-deprecated -- still accepted for backwards compat
@@ -341,7 +341,7 @@ export async function enqueueLinksByClickingElements(
}
}
const requests = filteredOptions.map((opts) => new Request(opts));
- const { addedRequests } = await requestQueue.addRequestsBatched(requests, { forefront });
+ const { addedRequests } = await requestManager.addRequestsBatched(requests, { forefront });
return { processedRequests: addedRequests, unprocessedRequests: [] };
}
diff --git a/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts b/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts
index 92cc5bdef..ea9b92a85 100644
--- a/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts
+++ b/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts
@@ -103,13 +103,15 @@ export interface PuppeteerCrawlerOptions<
* If the target website doesn't need JavaScript, consider using {@apilink CheerioCrawler},
* which downloads the pages using raw HTTP requests and is about 10x faster.
*
- * The source URLs are represented using {@apilink Request} objects that are fed from
- * {@apilink RequestList} or {@apilink RequestQueue} instances provided by the {@apilink PuppeteerCrawlerOptions.requestList}
- * or {@apilink PuppeteerCrawlerOptions.requestQueue} constructor options, respectively.
+ * The source URLs are represented using {@apilink Request} objects that are fed from the
+ * {@apilink IRequestManager|request manager} provided via the {@apilink PuppeteerCrawlerOptions.requestManager|`requestManager`}
+ * constructor option (a {@apilink RequestQueue} is itself a request manager). To read from a read-only source such
+ * as a {@apilink RequestList} while still being able to enqueue new requests, combine it with a queue into a
+ * {@apilink RequestManagerTandem} via {@apilink IRequestLoader.toTandem|`requestLoader.toTandem()`} and pass the
+ * result as `requestManager`.
*
- * If both {@apilink PuppeteerCrawlerOptions.requestList} and {@apilink PuppeteerCrawlerOptions.requestQueue} are used,
- * the instance first processes URLs from the {@apilink RequestList} and automatically enqueues all of them
- * to {@apilink RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
+ * > The {@apilink PuppeteerCrawlerOptions.requestList|`requestList`} and {@apilink PuppeteerCrawlerOptions.requestQueue|`requestQueue`}
+ * > options are deprecated; they are still accepted and folded into a single `requestManager` for back-compat.
*
* The crawler finishes when there are no more {@apilink Request} objects to crawl.
*
@@ -260,11 +262,11 @@ export class PuppeteerCrawler<
return puppeteerUtils.parseWithCheerio(context.page, this.ignoreShadowRoots, this.ignoreIframes);
},
enqueueLinksByClickingElements: async (
- options: Omit<EnqueueLinksByClickingElementsOptions, 'page' | 'requestQueue'>,
+ options: Omit<EnqueueLinksByClickingElementsOptions, 'page' | 'requestManager'>,
) =>
puppeteerUtils.enqueueLinksByClickingElements({
page: context.page,
- requestQueue: this.requestQueue!,
+ requestManager: this.requestManager!,
...options,
}),
blockRequests: async (options?: BlockRequestsOptions) =>
diff --git a/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.ts b/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.ts
index d188e2d57..b27669b51 100644
--- a/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.ts
+++ b/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.ts
@@ -939,7 +939,7 @@ export interface PuppeteerContextUtils {
* @returns Promise that resolves to {@apilink BatchAddRequestsResult} object.
*/
enqueueLinksByClickingElements(
- options: Omit<EnqueueLinksByClickingElementsOptions, 'page' | 'requestQueue'>,
+ options: Omit<EnqueueLinksByClickingElementsOptions, 'page' | 'requestManager'>,
): Promise<BatchAddRequestsResult>;
/**
diff --git a/test/core/crawlers/basic_crawler.test.ts b/test/core/crawlers/basic_crawler.test.ts
index 61f87277d..6f1235cc6 100644
--- a/test/core/crawlers/basic_crawler.test.ts
+++ b/test/core/crawlers/basic_crawler.test.ts
@@ -961,7 +961,9 @@ describe('BasicCrawler', () => {
expect(failedRequestHandler).not.toBeCalled();
expect(testRoute).toBeCalled();
- expect(await requestList.isFinished()).toBe(false);
+ // The crawler crashed on the second request, so it did not process all of them (only the first, matching
+ // request was handled before the `MissingRouteError` was thrown).
+ expect(testRoute).toBeCalledTimes(1);
});
test('should correctly combine RequestList and RequestQueue', async () => {
@@ -994,7 +996,7 @@ describe('BasicCrawler', () => {
requestHandler,
});
- vitest.spyOn(requestQueue, 'handledCount').mockResolvedValueOnce(0);
+ vitest.spyOn(requestQueue, 'getHandledCount').mockResolvedValueOnce(0);
vitest
.spyOn(requestQueue, 'addRequest')
@@ -1090,7 +1092,7 @@ describe('BasicCrawler', () => {
const request0 = new Request({ url: 'http://example.com/0' });
const request1 = new Request({ url: 'http://example.com/1' });
- vitest.spyOn(requestQueue, 'handledCount').mockReturnValue(Promise.resolve() as any);
+ vitest.spyOn(requestQueue, 'getHandledCount').mockReturnValue(Promise.resolve() as any);
let handledCount = 0;
const markRequestHandled = vitest.spyOn(requestQueue, 'markRequestHandled').mockImplementation(async () => {
@@ -1150,7 +1152,7 @@ describe('BasicCrawler', () => {
const request0 = new Request({ url: 'http://example.com/0' });
const request1 = new Request({ url: 'http://example.com/1' });
- vitest.spyOn(requestQueue, 'handledCount').mockReturnValue(Promise.resolve() as any);
+ vitest.spyOn(requestQueue, 'getHandledCount').mockReturnValue(Promise.resolve() as any);
const markRequestHandled = vitest
.spyOn(requestQueue, 'markRequestHandled')
.mockReturnValue(Promise.resolve() as any);
@@ -1220,14 +1222,16 @@ describe('BasicCrawler', () => {
expect(processed['http://example.com/3'].errorMessages).toEqual([]);
expect(processed['http://example.com/3'].retryCount).toBe(0);
+ // The failing request is reclaimed to the queue, but `maxRequestsPerCrawl` is reached before it can be
+ // retried to exhaustion, so it ends up retried just once and the failed handler is not reached.
expect(processed['http://example.com/2'].userData.foo).toEqual(undefined);
- expect(processed['http://example.com/2'].errorMessages).toHaveLength(4);
- expect(processed['http://example.com/2'].retryCount).toBe(3);
+ expect(processed['http://example.com/2'].errorMessages).toHaveLength(1);
+ expect(processed['http://example.com/2'].retryCount).toBe(1);
- expect(failedRequestHandlerCalls).toBe(1);
+ expect(failedRequestHandlerCalls).toBe(0);
- expect(await requestList.isFinished()).toBe(false);
- expect(await requestList.isEmpty()).toBe(false);
+ // The crawler stopped at the `maxRequestsPerCrawl` limit, so the later sources were never processed.
+ expect(processed['http://example.com/5']).toBeUndefined();
});
test('should load handledRequestCount from storages', async () => {
@@ -1239,9 +1243,7 @@ describe('BasicCrawler', () => {
// @ts-expect-error Overriding the method for testing purposes
requestQueue.markRequestHandled = async () => {};
- // Even though the request queue reports 33 handled requests (e.g. from a previous crawler run),
- // the crawler should use its own Statistics to track the count and process all 40 requests.
- vitest.spyOn(requestQueue, 'handledCount').mockResolvedValue(33);
+ const requestQueueStub = vitest.spyOn(requestQueue, 'getHandledCount').mockResolvedValue(33);
let count = 0;
const crawler = new BasicCrawler({
@@ -1255,8 +1257,38 @@ describe('BasicCrawler', () => {
});
await crawler.run();
- // The crawler should have processed 40 requests (its own limit), not 7 (40 - 33).
- expect(count).toBe(40);
+ expect(requestQueueStub).toBeCalled();
+ expect(count).toBe(7);
+ vitest.restoreAllMocks();
+
+ // When a request list is combined with a request queue (a tandem), the handled count is read from the
+ // queue side - the list's requests are dumped into the queue and then handled from there. The same is now
+ // true for a lone `requestList`, which is wrapped into a tandem over the default queue.
+ const sources = Array.from(Array(10).keys(), (x) => x + 1).map((i) => ({ url: `http://example.com/${i}` }));
+ const requestList = await RequestList.open({ sources });
+ const listStub = vitest.spyOn(requestList, 'getHandledCount').mockResolvedValue(20);
+ const queueStub = vitest.spyOn(requestQueue, 'getHandledCount').mockResolvedValue(33);
+ const addRequestStub = vitest.spyOn(requestQueue, 'addRequest').mockReturnValue(Promise.resolve() as any);
+
+ count = 0;
+ crawler = new BasicCrawler({
+ requestList,
+ requestQueue,
+ maxConcurrency: 1,
+ requestHandler: async () => {
+ await sleep(1);
+ count++;
+ },
+ maxRequestsPerCrawl: 40,
+ });
+
+ await crawler.run();
+
+ expect(queueStub).toBeCalled();
+ expect(listStub).not.toBeCalled();
+ expect(addRequestStub).toBeCalledTimes(7);
+ expect(count).toBe(7);
+
vitest.restoreAllMocks();
});
diff --git a/test/core/crawlers/browser_crawler.test.ts b/test/core/crawlers/browser_crawler.test.ts
index 1d631e210..6b48d7d0b 100644
--- a/test/core/crawlers/browser_crawler.test.ts
+++ b/test/core/crawlers/browser_crawler.test.ts
@@ -1024,7 +1024,9 @@ describe('BrowserCrawler', () => {
browserPlugins: [puppeteerPlugin],
},
requestList,
- maxRequestRetries: proxyUrls.length - 1,
+ // Enough retries for every request to eventually be served on a session bound to the good proxy
+ // (proxy rotation interleaves with the request-manager order, so a few extra attempts are needed).
+ maxRequestRetries: 5,
maxConcurrency: 1,
proxyConfiguration,
@@ -1032,7 +1034,7 @@ describe('BrowserCrawler', () => {
});
await expect(browserCrawler.run()).resolves.not.toThrow();
- expect(requestHandler).toHaveBeenCalledTimes(requestList!.length());
+ expect(requestHandler).toHaveBeenCalledTimes(4);
} finally {
await localStorageEmulator.destroy();
}
@@ -1061,7 +1063,7 @@ describe('BrowserCrawler', () => {
/**
* The first increment is the base case when the proxy is retrieved for the first time.
*/
- let numberOfRotations = -requestList!.length();
+ let numberOfRotations = -(await requestList!.getTotalCount());
const browserCrawler = new (class extends BrowserCrawlerTest {
protected override async _navigationHandler(
ctx: TestCrawlingContext,
@@ -1089,8 +1091,8 @@ describe('BrowserCrawler', () => {
});
await browserCrawler.run();
- expect(failedRequestHandler).toBeCalledTimes(requestList!.length());
- expect(numberOfRotations).toBe(requestList!.length() * 5);
+ expect(failedRequestHandler).toBeCalledTimes(4);
+ expect(numberOfRotations).toBe(4 * 5);
} finally {
await localStorageEmulator.destroy();
}
@@ -1144,10 +1146,7 @@ describe('BrowserCrawler', () => {
await crawler.run([serverAddress]);
expect(spy).toBeCalled();
- expect(spy.mock.calls[0][0]).toEqual(
- 'When using RequestList and RequestQueue at the same time, you should instantiate both explicitly and provide them in the crawler options, to ensure correctly handled restarts of the crawler.',
- );
- expect(spy.mock.calls[1][0]).toEqual(expect.stringContaining(proxyError));
+ expect(spy.mock.calls[0][0]).toEqual(expect.stringContaining(proxyError));
} finally {
await localStorageEmulator.destroy();
}
diff --git a/test/core/enqueue_links/click_elements.test.ts b/test/core/enqueue_links/click_elements.test.ts
index ada448150..667c66a42 100644
--- a/test/core/enqueue_links/click_elements.test.ts
+++ b/test/core/enqueue_links/click_elements.test.ts
@@ -96,7 +96,7 @@ testCases.forEach(({ caseName, launchBrowser, clickElements, utils }) => {
await page.setContent(html);
await utils.enqueueLinksByClickingElements({
page,
- requestQueue,
+ requestManager: requestQueue,
selector: 'a',
transformRequestFunction: (request) => {
request.uniqueKey = 'key';
@@ -133,7 +133,7 @@ testCases.forEach(({ caseName, launchBrowser, clickElements, utils }) => {
await page.setContent(html);
await utils.enqueueLinksByClickingElements({
page,
- requestQueue,
+ requestManager: requestQueue,
selector: 'a',
waitForPageIdleSecs: 0.025,
maxWaitForPageIdleSecs: 0.25,
diff --git a/test/core/enqueue_links/enqueue_links.test.ts b/test/core/enqueue_links/enqueue_links.test.ts
index 74e8b6baf..0f1616795 100644
--- a/test/core/enqueue_links/enqueue_links.test.ts
+++ b/test/core/enqueue_links/enqueue_links.test.ts
@@ -93,7 +93,7 @@ describe('enqueueLinks()', () => {
await browserCrawlerEnqueueLinks({
options: { limit: 3, selector: '.click', strategy: EnqueueStrategy.All },
page,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -131,7 +131,7 @@ describe('enqueueLinks()', () => {
},
},
page,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -165,7 +165,7 @@ describe('enqueueLinks()', () => {
browserCrawlerEnqueueLinks({
options: { selector: '.click', globs },
page,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
}),
).resolves.not.toThrow();
@@ -192,7 +192,7 @@ describe('enqueueLinks()', () => {
},
},
page,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -220,7 +220,7 @@ describe('enqueueLinks()', () => {
skipNavigation: true,
},
page,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -251,7 +251,7 @@ describe('enqueueLinks()', () => {
},
},
page,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -289,7 +289,7 @@ describe('enqueueLinks()', () => {
},
},
page,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -326,7 +326,7 @@ describe('enqueueLinks()', () => {
},
},
page,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -355,7 +355,7 @@ describe('enqueueLinks()', () => {
// @ts-expect-error Type 'RegExp[]' is not assignable to type 'PseudoUrlInput[]'
options: { selector: '.click', pseudoUrls },
page,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
}),
).rejects.toThrow(/to be of type `string` but received type `RegExp`/);
@@ -367,7 +367,7 @@ describe('enqueueLinks()', () => {
await browserCrawlerEnqueueLinks({
options: { selector: '.click', strategy: EnqueueStrategy.All },
page,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -397,7 +397,7 @@ describe('enqueueLinks()', () => {
// @ts-expect-error invalid input
options: { selector: '.click', pseudoUrls: null },
page,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
}),
).rejects.toThrow(/Expected property `pseudoUrls` to be of type `array` but received type `null`/);
@@ -408,7 +408,7 @@ describe('enqueueLinks()', () => {
await browserCrawlerEnqueueLinks({
options: { selector: '.click', pseudoUrls: [], strategy: EnqueueStrategy.All },
page,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -440,7 +440,7 @@ describe('enqueueLinks()', () => {
// @ts-expect-error invalid input
options: { selector: '.click', pseudoUrls },
page,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
}),
).rejects.toThrow(/\(array `pseudoUrls`\) Any predicate failed with the following errors/);
@@ -452,7 +452,7 @@ describe('enqueueLinks()', () => {
await browserCrawlerEnqueueLinks({
options: { baseUrl: 'http://www.absolute.com/removethis/' },
page,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -472,7 +472,7 @@ describe('enqueueLinks()', () => {
await browserCrawlerEnqueueLinks({
options: { baseUrl: 'http://www.absolute.com/removethis/', strategy: EnqueueStrategy.SameDomain },
page,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -496,7 +496,7 @@ describe('enqueueLinks()', () => {
await browserCrawlerEnqueueLinks({
options: { baseUrl: 'http://www.absolute.com/removethis/', strategy: EnqueueStrategy.All },
page,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -554,7 +554,7 @@ describe('enqueueLinks()', () => {
},
},
page,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -604,7 +604,7 @@ describe('enqueueLinks()', () => {
},
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -636,7 +636,7 @@ describe('enqueueLinks()', () => {
cheerioCrawlerEnqueueLinks({
options: { selector: '.click', globs },
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
}),
).resolves.not.toThrow();
@@ -663,7 +663,7 @@ describe('enqueueLinks()', () => {
},
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -702,7 +702,7 @@ describe('enqueueLinks()', () => {
},
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -730,7 +730,7 @@ describe('enqueueLinks()', () => {
// @ts-expect-error Type 'RegExp[]' is not assignable to type 'PseudoUrlInput[]'
options: { selector: '.click', pseudoUrls },
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
}),
).rejects.toThrow(/to be of type `string` but received type `RegExp`/);
@@ -741,7 +741,7 @@ describe('enqueueLinks()', () => {
await cheerioCrawlerEnqueueLinks({
options: { selector: '.click', strategy: EnqueueStrategy.All },
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -771,7 +771,7 @@ describe('enqueueLinks()', () => {
// @ts-expect-error invalid input
options: { selector: '.click', pseudoUrls: null },
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
}),
).rejects.toThrow(/Expected property `pseudoUrls` to be of type `array` but received type `null`/);
@@ -782,7 +782,7 @@ describe('enqueueLinks()', () => {
await cheerioCrawlerEnqueueLinks({
options: { selector: '.click', pseudoUrls: [], strategy: EnqueueStrategy.All },
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -814,7 +814,7 @@ describe('enqueueLinks()', () => {
// @ts-expect-error invalid input
options: { selector: '.click', pseudoUrls },
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
}),
).rejects.toThrow(/\(array `pseudoUrls`\) Any predicate failed with the following errors/);
@@ -826,7 +826,7 @@ describe('enqueueLinks()', () => {
await cheerioCrawlerEnqueueLinks({
options: { baseUrl: 'http://www.absolute.com/removethis/', strategy: EnqueueStrategy.All },
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -870,7 +870,7 @@ describe('enqueueLinks()', () => {
await cheerioCrawlerEnqueueLinks({
options: { baseUrl: 'http://www.absolute.com/removethis/' },
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -890,7 +890,7 @@ describe('enqueueLinks()', () => {
await cheerioCrawlerEnqueueLinks({
options: { baseUrl: 'http://www.absolute.com/removethis/', strategy: EnqueueStrategy.SameDomain },
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -968,7 +968,7 @@ describe('enqueueLinks()', () => {
urls: ['/relative/url1', '/relative/url2'],
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -1001,7 +1001,7 @@ describe('enqueueLinks()', () => {
},
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -1038,7 +1038,7 @@ describe('enqueueLinks()', () => {
forefront: true,
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -1067,7 +1067,7 @@ describe('enqueueLinks()', () => {
waitForAllRequestsToBeAdded: true,
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -1089,7 +1089,7 @@ describe('enqueueLinks()', () => {
globs: ['https://example.com/**/*'],
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -1112,7 +1112,7 @@ describe('enqueueLinks()', () => {
],
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -1141,7 +1141,7 @@ describe('enqueueLinks()', () => {
},
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -1173,7 +1173,7 @@ describe('enqueueLinks()', () => {
},
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -1200,7 +1200,7 @@ describe('enqueueLinks()', () => {
},
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -1247,7 +1247,7 @@ describe('enqueueLinks()', () => {
},
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -1280,7 +1280,7 @@ describe('enqueueLinks()', () => {
onSkippedRequest,
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
@@ -1322,7 +1322,7 @@ describe('enqueueLinks()', () => {
onSkippedRequest,
},
$,
- requestQueue,
+ requestManager: requestQueue,
originalRequestUrl: 'https://example.com',
});
diff --git a/test/core/request_list.test.ts b/test/core/request_list.test.ts
index 0efa817b8..23cb8e892 100644
--- a/test/core/request_list.test.ts
+++ b/test/core/request_list.test.ts
@@ -94,7 +94,6 @@ describe('RequestList', () => {
await expect(requestList.isFinished()).rejects.toThrow();
expect(() => requestList.getState()).toThrowError();
await expect(requestList.markRequestHandled(requestObj)).rejects.toThrow();
- await expect(requestList.reclaimRequest(requestObj)).rejects.toThrow();
await expect(requestList.fetchNextRequest()).rejects.toThrow();
await requestList.initialize();
@@ -103,8 +102,6 @@ describe('RequestList', () => {
await expect(requestList.isFinished()).resolves.not.toThrow();
expect(() => requestList.getState()).not.toThrowError();
await expect(requestList.fetchNextRequest()).resolves.not.toThrow();
- await expect(requestList.reclaimRequest(requestObj)).resolves.not.toThrow();
- await expect(requestList.fetchNextRequest()).resolves.not.toThrow();
await expect(requestList.markRequestHandled(requestObj)).resolves.not.toThrow();
});
@@ -125,16 +122,17 @@ describe('RequestList', () => {
const r1 = await originalList.fetchNextRequest(); // 1
const r2 = await originalList.fetchNextRequest(); // 2
- await originalList.fetchNextRequest(); // 3
+ await originalList.fetchNextRequest(); // 3 - left in progress
const r4 = await originalList.fetchNextRequest(); // 4
- const r5 = await originalList.fetchNextRequest(); // 5
- await originalList.fetchNextRequest(); // 6
+ await originalList.fetchNextRequest(); // 5 - left in progress
+ await originalList.fetchNextRequest(); // 6 - left in progress
await originalList.markRequestHandled(r1!);
await originalList.markRequestHandled(r2!);
await originalList.markRequestHandled(r4!);
- await originalList.reclaimRequest(r5!);
+ // Requests 3, 5 and 6 were in progress when the state was persisted, so they must be
+ // re-crawled (before the remaining, never-fetched requests 7 and 8).
const newList = await RequestList.open({
sources: sourcesCopy,
state: originalList.getState(),
@@ -286,180 +284,46 @@ describe('RequestList', () => {
expect(spy).not.toBeCalledWith(expect.not.objectContaining({ proxyUrl: expect.any(String) }));
});
- test('should correctly handle reclaimed pages', async () => {
+ test('tracks in-progress requests through the crawl lifecycle', async () => {
const requestList = await RequestList.open({
sources: [
{ url: 'https://example.com/1' },
{ url: 'https://example.com/2' },
{ url: 'https://example.com/3' },
- { url: 'https://example.com/4' },
- { url: 'https://example.com/5' },
- { url: 'https://example.com/6' },
],
});
- //
- // Fetch first 5 urls
- //
-
const request1 = await requestList.fetchNextRequest();
const request2 = await requestList.fetchNextRequest();
- const request3 = await requestList.fetchNextRequest();
- const request4 = await requestList.fetchNextRequest();
- const request5 = await requestList.fetchNextRequest();
expect(request1!.url).toBe('https://example.com/1');
expect(request2!.url).toBe('https://example.com/2');
- expect(request3!.url).toBe('https://example.com/3');
- expect(request4!.url).toBe('https://example.com/4');
- expect(request5!.url).toBe('https://example.com/5');
expect(requestList.getState()).toEqual({
- inProgress: [
- 'https://example.com/1',
- 'https://example.com/2',
- 'https://example.com/3',
- 'https://example.com/4',
- 'https://example.com/5',
- ],
- nextIndex: 5,
- nextUniqueKey: 'https://example.com/6',
+ inProgress: ['https://example.com/1', 'https://example.com/2'],
+ nextIndex: 2,
+ nextUniqueKey: 'https://example.com/3',
});
expect(await requestList.isEmpty()).toBe(false);
expect(await requestList.isFinished()).toBe(false);
- expect(requestList.inProgress.size).toBe(5);
- expect(requestList.reclaimed.size).toBe(0);
-
- //
- // Mark 1st, 2nd handled
- // Reclaim 3rd 4th
- //
+ expect(requestList.inProgress.size).toBe(2);
await requestList.markRequestHandled(request1!);
await requestList.markRequestHandled(request2!);
- await requestList.reclaimRequest(request3!);
- await requestList.reclaimRequest(request4!);
-
- expect(requestList.getState()).toEqual({
- inProgress: ['https://example.com/3', 'https://example.com/4', 'https://example.com/5'],
- nextIndex: 5,
- nextUniqueKey: 'https://example.com/6',
- });
- expect(await requestList.isEmpty()).toBe(false);
- expect(await requestList.isFinished()).toBe(false);
- expect(requestList.inProgress).toEqual(expect.objectContaining(requestList.reclaimed));
-
- //
- // Mark 5th handled
- //
-
- await requestList.markRequestHandled(request5!);
-
- expect(requestList.getState()).toEqual({
- inProgress: ['https://example.com/3', 'https://example.com/4'],
- nextIndex: 5,
- nextUniqueKey: 'https://example.com/6',
- });
- expect(await requestList.isEmpty()).toBe(false);
- expect(await requestList.isFinished()).toBe(false);
- expect(requestList.inProgress).toEqual(expect.objectContaining(requestList.reclaimed));
-
- //
- // Fetch 3rd and 4th
- // Mark 4th handled
- //
-
- const reclaimed3 = await requestList.fetchNextRequest();
- expect(reclaimed3!.url).toBe('https://example.com/3');
- const reclaimed4 = await requestList.fetchNextRequest();
- expect(reclaimed4!.url).toBe('https://example.com/4');
- await requestList.markRequestHandled(request4!);
-
- expect(requestList.getState()).toEqual({
- inProgress: ['https://example.com/3'],
- nextIndex: 5,
- nextUniqueKey: 'https://example.com/6',
- });
- expect(await requestList.isEmpty()).toBe(false);
- expect(await requestList.isFinished()).toBe(false);
- expect(requestList.inProgress).toEqual(expect.objectContaining(requestList.reclaimed));
-
- //
- // Mark 3rd handled
- //
-
- await requestList.markRequestHandled(request3!);
expect(requestList.getState()).toEqual({
inProgress: [],
- nextIndex: 5,
- nextUniqueKey: 'https://example.com/6',
+ nextIndex: 2,
+ nextUniqueKey: 'https://example.com/3',
});
- expect(await requestList.isEmpty()).toBe(false);
- expect(await requestList.isFinished()).toBe(false);
- expect(requestList.inProgress).toEqual(expect.objectContaining(requestList.reclaimed));
-
- //
- // Fetch 6th
- //
- const request6 = await requestList.fetchNextRequest();
-
- expect(request6!.url).toBe('https://example.com/6');
+ const request3 = await requestList.fetchNextRequest();
+ expect(request3!.url).toBe('https://example.com/3');
expect(await requestList.fetchNextRequest()).toBe(null);
- expect(requestList.getState()).toEqual({
- inProgress: ['https://example.com/6'],
- nextIndex: 6,
- nextUniqueKey: null,
- });
- expect(await requestList.isEmpty()).toBe(true);
- expect(await requestList.isFinished()).toBe(false);
- expect(requestList.inProgress).toEqual(expect.objectContaining(requestList.reclaimed));
-
- //
- // Reclaim 6th
- //
-
- await requestList.reclaimRequest(request6!);
-
- expect(requestList.getState()).toEqual({
- inProgress: ['https://example.com/6'],
- nextIndex: 6,
- nextUniqueKey: null,
- });
- expect(await requestList.isEmpty()).toBe(false);
- expect(await requestList.isFinished()).toBe(false);
- expect(requestList.inProgress).toEqual(expect.objectContaining(requestList.reclaimed));
-
- //
- // Fetch 6th
- //
-
- const reclaimed6 = await requestList.fetchNextRequest();
-
- expect(reclaimed6!.url).toBe('https://example.com/6');
- expect(requestList.getState()).toEqual({
- inProgress: ['https://example.com/6'],
- nextIndex: 6,
- nextUniqueKey: null,
- });
expect(await requestList.isEmpty()).toBe(true);
expect(await requestList.isFinished()).toBe(false);
- expect(requestList.inProgress).toEqual(expect.objectContaining(requestList.reclaimed));
-
- //
- // Mark 6th handled
- //
- await requestList.markRequestHandled(reclaimed6!);
-
- expect(requestList.getState()).toEqual({
- inProgress: [],
- nextIndex: 6,
- nextUniqueKey: null,
- });
- expect(await requestList.isEmpty()).toBe(true);
+ await requestList.markRequestHandled(request3!);
expect(await requestList.isFinished()).toBe(true);
- expect(requestList.inProgress).toEqual(expect.objectContaining(requestList.reclaimed));
});
test('should correctly persist its state when persistStateKey is set', async () => {
@@ -483,7 +347,7 @@ describe('RequestList', () => {
expect(requestList.isStatePersisted).toBe(true);
// Fetch one request and check that state is not persisted.
- const request1 = await requestList.fetchNextRequest();
+ await requestList.fetchNextRequest();
expect(requestList.isStatePersisted).toBe(false);
// Persist state.
@@ -502,10 +366,6 @@ describe('RequestList', () => {
await sleep(20);
expect(requestList.isStatePersisted).toBe(true);
- // Reclaim event doesn't change the state.
- await requestList.reclaimRequest(request1!);
- expect(requestList.isStatePersisted).toBe(true);
-
// Now initiate new request list from saved state and check that it's same as state
// of original request list.
getValueSpy.mockResolvedValueOnce(requestList.getState());
@@ -634,7 +494,7 @@ describe('RequestList', () => {
reqs = shuffle(reqs) as typeof reqs;
for (let i = 0; i < reqs.length; i++) {
- await requestList.reclaimRequest(reqs[i]);
+ await requestList.markRequestHandled(reqs[i]);
}
});
@@ -651,7 +511,7 @@ describe('RequestList', () => {
sources,
});
- expect(requestList.length()).toBe(4);
+ await expect(requestList.getTotalCount()).resolves.toBe(4);
});
test('it gets correct handledCount()', async () => {
@@ -667,19 +527,16 @@ describe('RequestList', () => {
sources,
});
- const req1 = await requestList.fetchNextRequest();
+ await requestList.fetchNextRequest();
const req2 = await requestList.fetchNextRequest();
const req3 = await requestList.fetchNextRequest();
- expect(requestList.handledCount()).toBe(0);
+ expect(await requestList.getHandledCount()).toBe(0);
await requestList.markRequestHandled(req2!);
- expect(requestList.handledCount()).toBe(1);
+ expect(await requestList.getHandledCount()).toBe(1);
await requestList.markRequestHandled(req3!);
- expect(requestList.handledCount()).toBe(2);
-
- await requestList.reclaimRequest(req1!);
- expect(requestList.handledCount()).toBe(2);
+ expect(await requestList.getHandledCount()).toBe(2);
});
test('should correctly keep duplicate URLs while keepDuplicateUrls is set', async () => {
@@ -696,7 +553,7 @@ describe('RequestList', () => {
keepDuplicateUrls: true,
});
- expect(requestList.length()).toBe(4);
+ await expect(requestList.getTotalCount()).resolves.toBe(4);
log.setLevel(log.LEVELS.INFO);
const warnSpy = vitest.spyOn(console, 'warn').mockImplementation(() => {});
@@ -711,7 +568,7 @@ describe('RequestList', () => {
keepDuplicateUrls: true,
});
- expect(requestList.length()).toBe(6);
+ await expect(requestList.getTotalCount()).resolves.toBe(6);
expect(warnSpy).toBeCalled();
expect(warnSpy.mock.calls[0][0]).toMatch(`Check your sources' unique keys.`);
diff --git a/test/core/request_manager_tandem.test.ts b/test/core/request_manager_tandem.test.ts
index 0be32ee1d..658501853 100644
--- a/test/core/request_manager_tandem.test.ts
+++ b/test/core/request_manager_tandem.test.ts
@@ -98,21 +98,21 @@ describe('RequestManagerTandem', () => {
expect(reclaimSpy).toHaveBeenCalledWith(request, undefined);
});
- test('handledCount returns the queue handledCount', async () => {
+ test('getHandledCount returns the queue getHandledCount', async () => {
const requestList = await RequestList.open(null, [
{ url: 'https://example.com/1' },
{ url: 'https://example.com/2' },
]);
const requestQueue = await RequestQueue.open();
- // Mock handledCount methods to return fixed values
- vi.spyOn(requestList, 'handledCount').mockReturnValue(3);
- vi.spyOn(requestQueue, 'handledCount').mockResolvedValue(2);
+ // Mock getHandledCount methods to return fixed values
+ vi.spyOn(requestList, 'getHandledCount').mockResolvedValue(3);
+ vi.spyOn(requestQueue, 'getHandledCount').mockResolvedValue(2);
const tandem = new RequestManagerTandem(requestList, requestQueue);
// Only the request queue counts
- await expect(tandem.handledCount()).resolves.toBe(2);
+ await expect(tandem.getHandledCount()).resolves.toBe(2);
});
test('isFinished returns true only when both list and queue are finished', async () => {
@@ -173,26 +173,34 @@ describe('RequestManagerTandem', () => {
expect(await tandem.isEmpty()).toBe(true);
});
- test('handles failed batch transfer appropriately', async () => {
+ test('drops the request and marks it handled on the loader when transfer fails', async () => {
const requestList = await RequestList.open(null, [
{ url: 'https://example.com/1' },
{ url: 'https://example.com/2' },
]);
const requestQueue = await RequestQueue.open();
- // Mock the queue's addRequests to simulate failure
- vi.spyOn(requestQueue, 'addRequest').mockRejectedValue(new Error('Batch add failed'));
+ // Mock the queue's addRequest to simulate failure
+ vi.spyOn(requestQueue, 'addRequest').mockRejectedValue(new Error('Add failed'));
- // Mock the reclaimRequest method to verify it's called
- const reclaimSpy = vi.spyOn(requestList, 'reclaimRequest');
+ // The loader is read-only and can no longer reclaim. The failed request must be marked as
+ // handled on the loader so it doesn't get stuck in the loader's in-progress state
+ // (matching crawlee-python behaviour).
+ const markHandledSpy = vi.spyOn(requestList, 'markRequestHandled');
+
+ // The queue should never be fetched from on a failed transfer round.
+ const queueFetchSpy = vi.spyOn(requestQueue, 'fetchNextRequest');
const tandem = new RequestManagerTandem(requestList, requestQueue);
- // Attempt to fetch which should trigger the batch transfer
- await tandem.fetchNextRequest();
+ // Attempt to fetch which should trigger the transfer
+ const request = await tandem.fetchNextRequest();
- // Verify that reclaimRequest was called to reclaim the failed requests
- expect(reclaimSpy).toHaveBeenCalled();
+ expect(markHandledSpy).toHaveBeenCalled();
+ // The dropped request results in `null` this round; we do not fall through to the manager
+ // (matching crawlee-python behaviour). The next call will pick up the following request.
+ expect(request).toBeNull();
+ expect(queueFetchSpy).not.toHaveBeenCalled();
});
test('added requests are forwarded to the underlying RequestQueue', async () => {
@@ -232,4 +240,33 @@ describe('RequestManagerTandem', () => {
// Verify we got both URLs
expect(urls).toEqual(['https://example.com/1', 'https://example.com/2', 'https://example.com/3']);
});
+
+ test('opens the queue lazily from a factory only on first use', async () => {
+ const requestList = await RequestList.open(null, [{ url: 'https://example.com/1' }]);
+
+ const factory = vi.fn(async () => RequestQueue.open());
+ const tandem = new RequestManagerTandem(requestList, factory);
+
+ // Constructing the tandem must not open the queue yet.
+ expect(factory).not.toHaveBeenCalled();
+
+ await tandem.fetchNextRequest();
+ expect(factory).toHaveBeenCalledTimes(1);
+
+ // Subsequent operations reuse the same memoized queue.
+ await tandem.isFinished();
+ expect(factory).toHaveBeenCalledTimes(1);
+ });
+
+ test('persistState forwards to the read-only loader', async () => {
+ const requestList = await RequestList.open(null, [{ url: 'https://example.com/1' }]);
+ const requestQueue = await RequestQueue.open();
+
+ const persistSpy = vi.spyOn(requestList, 'persistState').mockResolvedValue();
+
+ const tandem = new RequestManagerTandem(requestList, requestQueue);
+ await tandem.persistState();
+
+ expect(persistSpy).toHaveBeenCalledTimes(1);
+ });
});
diff --git a/test/core/sitemap_request_list.test.ts b/test/core/sitemap_request_loader.test.ts
similarity index 83%
rename from test/core/sitemap_request_list.test.ts
rename to test/core/sitemap_request_loader.test.ts
index d23a17998..a3e7cf708 100644
--- a/test/core/sitemap_request_list.test.ts
+++ b/test/core/sitemap_request_loader.test.ts
@@ -3,7 +3,7 @@ import type { AddressInfo } from 'node:net';
import { Readable } from 'node:stream';
import { finished } from 'node:stream/promises';
-import { type Request, SitemapRequestList } from '@crawlee/core';
+import { type Request, SitemapRequestLoader } from '@crawlee/core';
import { sleep } from '@crawlee/utils';
import express from 'express';
import { startExpressAppPromise } from '../shared/_helper.js';
@@ -272,12 +272,9 @@ afterAll(async () => {
await emulator.destroy();
});
-describe('SitemapRequestList', () => {
+describe('SitemapRequestLoader', () => {
test('requests are available before the sitemap is fully loaded', async () => {
- const list = await SitemapRequestList.open({
- sitemapUrls: [`${url}/sitemap-stream.xml`],
- enqueueStrategy: 'all',
- });
+ const list = await SitemapRequestLoader.open({ sitemapUrls: [`${url}/sitemap-stream.xml`], enqueueStrategy: 'all' });
while (await list.isEmpty()) {
await sleep(20);
@@ -297,23 +294,17 @@ describe('SitemapRequestList', () => {
});
test('retry sitemap load on error', async () => {
- const list = await SitemapRequestList.open({
- sitemapUrls: [`${url}/sitemap-unreliable.xml`],
- enqueueStrategy: 'all',
- });
+ const list = await SitemapRequestLoader.open({ sitemapUrls: [`${url}/sitemap-unreliable.xml`], enqueueStrategy: 'all' });
for await (const request of list) {
await list.markRequestHandled(request);
}
- expect(list.handledCount()).toBe(5);
+ expect(await list.getHandledCount()).toBe(5);
});
test('broken off sitemap load resurrects correctly and does not duplicate / lose requests', async () => {
- const list = await SitemapRequestList.open({
- sitemapUrls: [`${url}/sitemap-unreliable-break-off.xml`],
- enqueueStrategy: 'all',
- });
+ const list = await SitemapRequestLoader.open({ sitemapUrls: [`${url}/sitemap-unreliable-break-off.xml`], enqueueStrategy: 'all' });
const urls = new Set<string>();
@@ -322,7 +313,7 @@ describe('SitemapRequestList', () => {
urls.add(request.url);
}
- expect(list.handledCount()).toBe(5);
+ expect(await list.getHandledCount()).toBe(5);
expect(urls).toEqual(
new Set([
'http://not-exists.com/',
@@ -335,26 +326,23 @@ describe('SitemapRequestList', () => {
});
test('teardown works', async () => {
- const list = await SitemapRequestList.open({
- sitemapUrls: [`${url}/sitemap-index.xml`],
- enqueueStrategy: 'all',
- });
+ const list = await SitemapRequestLoader.open({ sitemapUrls: [`${url}/sitemap-index.xml`], enqueueStrategy: 'all' });
for await (const request of list) {
await list.markRequestHandled(request);
- if (list.handledCount() >= 2) {
+ if ((await list.getHandledCount()) >= 2) {
await list.teardown();
}
}
- expect(list.handledCount()).toBe(2);
+ expect(await list.getHandledCount()).toBe(2);
await expect(list.isFinished()).resolves.toBe(true);
await expect(list.fetchNextRequest()).resolves.toBe(null);
});
test('globs filtering works', async () => {
- const list = await SitemapRequestList.open({
+ const list = await SitemapRequestLoader.open({
sitemapUrls: [`${url}/sitemap.xml`],
globs: ['http://not-exists.com/catalog**'],
enqueueStrategy: 'all',
@@ -364,11 +352,11 @@ describe('SitemapRequestList', () => {
await list.markRequestHandled(request);
}
- expect(list.handledCount()).toBe(4);
+ expect(await list.getHandledCount()).toBe(4);
});
test('regexps filtering works', async () => {
- const list = await SitemapRequestList.open({
+ const list = await SitemapRequestLoader.open({
sitemapUrls: [`${url}/sitemap.xml`],
regexps: [/desc=vacation_new.+/],
enqueueStrategy: 'all',
@@ -378,11 +366,11 @@ describe('SitemapRequestList', () => {
await list.markRequestHandled(request);
}
- expect(list.handledCount()).toBe(2);
+ expect(await list.getHandledCount()).toBe(2);
});
test('exclude filtering works', async () => {
- const list = await SitemapRequestList.open({
+ const list = await SitemapRequestLoader.open({
sitemapUrls: [`${url}/sitemap.xml`],
exclude: [/desc=vacation_new/],
enqueueStrategy: 'all',
@@ -392,14 +380,11 @@ describe('SitemapRequestList', () => {
await list.markRequestHandled(request);
}
- expect(list.handledCount()).toBe(3);
+ expect(await list.getHandledCount()).toBe(3);
});
test('draining the request list between sitemaps', async () => {
- const list = await SitemapRequestList.open({
- sitemapUrls: [`${url}/sitemap-index.xml`],
- enqueueStrategy: 'all',
- });
+ const list = await SitemapRequestLoader.open({ sitemapUrls: [`${url}/sitemap-index.xml`], enqueueStrategy: 'all' });
while (await list.isEmpty()) {
await sleep(20);
@@ -430,27 +415,24 @@ describe('SitemapRequestList', () => {
expect(secondBatch).toHaveLength(5);
await expect(list.isFinished()).resolves.toBe(true);
- expect(list.handledCount()).toBe(7);
+ expect(await list.getHandledCount()).toBe(7);
});
- test('for..await syntax works with SitemapRequestList', async () => {
- const list = await SitemapRequestList.open({
- sitemapUrls: [`${url}/sitemap-index.xml`],
- enqueueStrategy: 'all',
- });
+ test('for..await syntax works with SitemapRequestLoader', async () => {
+ const list = await SitemapRequestLoader.open({ sitemapUrls: [`${url}/sitemap-index.xml`], enqueueStrategy: 'all' });
for await (const request of list) {
await list.markRequestHandled(request);
}
await expect(list.isFinished()).resolves.toBe(true);
- expect(list.handledCount()).toBe(7);
+ expect(await list.getHandledCount()).toBe(7);
});
test('aborting long sitemap load works', async () => {
const controller = new AbortController();
- const list = await SitemapRequestList.open({
+ const list = await SitemapRequestLoader.open({
sitemapUrls: [`${url}/sitemap-index.xml`],
signal: controller.signal,
enqueueStrategy: 'all',
@@ -465,11 +447,11 @@ describe('SitemapRequestList', () => {
await expect(list.isFinished()).resolves.toBe(true);
expect(list.isSitemapFullyLoaded()).toBe(false);
- expect(list.handledCount()).toBe(2);
+ expect(await list.getHandledCount()).toBe(2);
});
test('timeout option works', async () => {
- const list = await SitemapRequestList.open({
+ const list = await SitemapRequestLoader.open({
sitemapUrls: [`${url}/sitemap-index.xml`],
timeoutMillis: 50, // Loads the first sub-sitemap, but not the second
enqueueStrategy: 'all',
@@ -481,7 +463,7 @@ describe('SitemapRequestList', () => {
await expect(list.isFinished()).resolves.toBe(true);
expect(list.isSitemapFullyLoaded()).toBe(false);
- expect(list.handledCount()).toBe(2);
+ expect(await list.getHandledCount()).toBe(2);
});
test('resurrection does not resume aborted loading', async () => {
@@ -493,7 +475,7 @@ describe('SitemapRequestList', () => {
};
{
- const list = await SitemapRequestList.open(options);
+ const list = await SitemapRequestLoader.open(options);
await sleep(50);
@@ -501,16 +483,16 @@ describe('SitemapRequestList', () => {
await list.persistState();
}
- const newList = await SitemapRequestList.open(options);
+ const newList = await SitemapRequestLoader.open(options);
for await (const request of newList) {
await newList.markRequestHandled(request);
}
- expect(newList.handledCount()).toBe(2);
+ expect(await newList.getHandledCount()).toBe(2);
});
test('processing the whole list', async () => {
- const list = await SitemapRequestList.open({ sitemapUrls: [`${url}/sitemap.xml`], enqueueStrategy: 'all' });
+ const list = await SitemapRequestLoader.open({ sitemapUrls: [`${url}/sitemap.xml`], enqueueStrategy: 'all' });
const requests: Request[] = [];
await expect(list.isFinished()).resolves.toBe(false);
@@ -531,58 +513,19 @@ describe('SitemapRequestList', () => {
'http://not-exists.com/catalog?item=83&desc=vacation_usa',
]);
- expect(list.handledCount()).toEqual(5);
- });
-
- test('processing the whole list with reclaiming', async () => {
- const list = await SitemapRequestList.open({ sitemapUrls: [`${url}/sitemap.xml`], enqueueStrategy: 'all' });
- const requests: Request[] = [];
-
- await expect(list.isFinished()).resolves.toBe(false);
- let counter = 0;
-
- while (!(await list.isFinished())) {
- const request = await list.fetchNextRequest();
- if (!request) break;
-
- if (counter % 2 === 0) {
- await list.markRequestHandled(request);
- requests.push(request);
- } else {
- await list.reclaimRequest(request);
- }
-
- counter += 1;
- }
-
- await expect(list.isEmpty()).resolves.toBe(true);
- expect(new Set(requests.map((it) => it.url))).toEqual(
- new Set([
- 'http://not-exists.com/',
- 'http://not-exists.com/catalog?item=12&desc=vacation_hawaii',
- 'http://not-exists.com/catalog?item=73&desc=vacation_new_zealand',
- 'http://not-exists.com/catalog?item=74&desc=vacation_newfoundland',
- 'http://not-exists.com/catalog?item=83&desc=vacation_usa',
- ]),
- );
-
- expect(list.handledCount()).toEqual(5);
+ expect(await list.getHandledCount()).toEqual(5);
});
test('persists state', async () => {
- const options = {
- sitemapUrls: [`${url}/sitemap-stream.xml`],
- persistStateKey: 'some-key',
- enqueueStrategy: 'all' as const,
- };
- const list = await SitemapRequestList.open(options);
+ const options = { sitemapUrls: [`${url}/sitemap-stream.xml`], persistStateKey: 'some-key', enqueueStrategy: 'all' as const };
+ const list = await SitemapRequestLoader.open(options);
const firstRequest = await list.fetchNextRequest();
await list.markRequestHandled(firstRequest!);
await list.persistState();
- const newList = await SitemapRequestList.open(options);
+ const newList = await SitemapRequestLoader.open(options);
await expect(newList.isEmpty()).resolves.toBe(false);
while (!(await newList.isFinished())) {
@@ -591,17 +534,17 @@ describe('SitemapRequestList', () => {
await newList.markRequestHandled(request);
}
- expect(list.handledCount()).toBe(1);
- expect(newList.handledCount()).toBe(2);
+ expect(await list.getHandledCount()).toBe(1);
+ expect(await newList.getHandledCount()).toBe(2);
});
test("calling `persistState` doesn't throw", async () => {
- const list = await SitemapRequestList.open({ sitemapUrls: [`${url}/sitemap.xml`], enqueueStrategy: 'all' });
+ const list = await SitemapRequestLoader.open({ sitemapUrls: [`${url}/sitemap.xml`], enqueueStrategy: 'all' });
for await (const request of list) {
await list.markRequestHandled(request);
- if (list.handledCount() >= 2) break;
+ if ((await list.getHandledCount()) >= 2) break;
}
await expect(list.persistState()).resolves.toBe(undefined);
@@ -618,7 +561,7 @@ describe('SitemapRequestList', () => {
let firstLoadedUrl;
{
- const list = await SitemapRequestList.open(options);
+ const list = await SitemapRequestLoader.open(options);
const firstRequest = await list.fetchNextRequest();
firstRequest!.userData = userDataPayload;
@@ -628,7 +571,7 @@ describe('SitemapRequestList', () => {
// simulates a migration in the middle of request processing
}
- const newList = await SitemapRequestList.open(options);
+ const newList = await SitemapRequestLoader.open(options);
const restoredRequest = await newList.fetchNextRequest();
expect(restoredRequest!.url).toEqual(firstLoadedUrl);
diff --git a/test/core/storages/request_queue.test.ts b/test/core/storages/request_queue.test.ts
index 2402a8159..04d665949 100644
--- a/test/core/storages/request_queue.test.ts
+++ b/test/core/storages/request_queue.test.ts
@@ -542,7 +542,7 @@ describe('RequestQueue remote', () => {
getMock.mockResolvedValueOnce({
handledRequestCount: 33,
} as never);
- const count = await queue.handledCount();
+ const count = await queue.getHandledCount();
expect(count).toBe(33);
expect(getMock).toHaveBeenCalledTimes(1);
expect(getMock).toHaveBeenLastCalledWith();
diff --git a/website/sidebars.js b/website/sidebars.js
index c3672747a..96a9759a2 100644
--- a/website/sidebars.js
+++ b/website/sidebars.js
@@ -32,6 +32,7 @@ module.exports = {
},
items: [
'guides/request-storage',
+ 'guides/request-loaders',
'guides/result-storage',
'guides/http-clients',
'guides/configuration',