Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
refactor!: Overhaul the StorageClient interface (#3570)
Jan Buchar6cbd3acc69ea
diff --git a/docs/examples/skip-navigation.ts b/docs/examples/skip-navigation.tsindex 679150e42..867fb4732 100644--- a/docs/examples/skip-navigation.ts+++ b/docs/examples/skip-navigation.ts@@ -1,7 +1,7 @@import { PlaywrightCrawler, KeyValueStore } from 'crawlee';// Create a key value store for all images we find-const imageStore = await KeyValueStore.open('images');+const imageStore = await KeyValueStore.open({ name: 'images' });const crawler = new PlaywrightCrawler({async requestHandler({ request, page, sendRequest }) {diff --git a/docs/guides/request_storage_queue_basic.ts b/docs/guides/request_storage_queue_basic.tsindex 66d3d3372..1555e5bbd 100644--- a/docs/guides/request_storage_queue_basic.ts+++ b/docs/guides/request_storage_queue_basic.ts@@ -11,7 +11,7 @@ await requestQueue.addRequests([]);// Open the named request queue-const namedRequestQueue = await RequestQueue.open('named-queue');+const namedRequestQueue = await RequestQueue.open({ name: 'named-queue' });// Remove the named request queueawait namedRequestQueue.drop();diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.mdindex f53dd8c46..24d304107 100644--- a/docs/upgrading/upgrading_v4.md+++ b/docs/upgrading/upgrading_v4.md@@ -269,6 +269,85 @@ const response = await sendRequest({ url: '...' }, { cookieJar: jar });The protected `HttpCrawler._applyCookies` method is removed. If you were overriding it in a subclass, move your logic to a `preNavigationHook` that sets cookies on `request.headers.Cookie` or on the `session` cookie jar directly.+## `StorageClient` interface simplified++The `StorageClient` interface (from `@crawlee/types`) has been redesigned to match the simplified architecture from Crawlee for Python. A new storage backend now needs **4 classes** instead of the previous 7.++### What changed++The three **collection client** interfaces have been removed:++- `DatasetCollectionClient`+- `KeyValueStoreCollectionClient`+- `RequestQueueCollectionClient`++Along with their associated types (`DatasetCollectionData`, `DatasetCollectionClientOptions`, and the `Dataset` interface from `@crawlee/types`).++The `StorageClient` interface changed from synchronous sub-client getters to **async factory methods**:++| Before (v3) | After (v4) |+|---|---|+| `client.dataset(id)` | `client.createDatasetClient({ id?, name? })` |+| `client.datasets().getOrCreate(name)` | _(absorbed into `createDatasetClient`)_ |+| `client.keyValueStore(id)` | `client.createKeyValueStoreClient({ id?, name? })` |+| `client.keyValueStores().getOrCreate(name)` | _(absorbed into `createKeyValueStoreClient`)_ |+| `client.requestQueue(id, opts)` | `client.createRequestQueueClient({ id?, name?, clientKey?, timeoutSecs? })` |+| `client.requestQueues().getOrCreate(name)` | _(absorbed into `createRequestQueueClient`)_ |++The `get()` method on `DatasetClient`, `KeyValueStoreClient`, and `RequestQueueClient` has been renamed to **`getMetadata()`**.++The high-level storage classes (`Dataset`, `KeyValueStore`, `RequestQueue`) now receive their sub-client directly in the constructor options instead of receiving a `StorageClient` and calling its methods.++### Removed `list()` method++The `list()` method on collection clients (e.g. `client.datasets().list()`) has no replacement. If you were using it to enumerate all storages, you will need to use the Apify API client directly.++### Migration guide++If you implemented a custom `StorageClient`, you need to:++1. Remove your `*CollectionClient` classes.+2. Replace the six getter methods (`dataset`, `datasets`, `keyValueStore`, `keyValueStores`, `requestQueue`, `requestQueues`) with three async factory methods (`createDatasetClient`, `createKeyValueStoreClient`, `createRequestQueueClient`). Each factory should handle both opening an existing storage and creating a new one.+3. Rename `get()` to `getMetadata()` on your `DatasetClient`, `KeyValueStoreClient`, and `RequestQueueClient` implementations.++## Storage `.open()` now also accepts `{ id?, name? }`++`Dataset.open()`, `KeyValueStore.open()`, and `RequestQueue.open()` previously accepted a single `idOrName?: string` parameter. This was ambiguous — callers couldn't express whether they were opening a storage by its ID or by name.++The first parameter now also accepts a `StorageIdentifier` object with separate `id` and `name` fields:++```ts+interface StorageIdentifier {+ id?: string;+ name?: string;+}+```++Passing a plain string still works — it is first looked up as an ID, and if no such storage exists, it is treated as a name (matching the v3 behavior):++```typescript+const dataset = await Dataset.open('my-dataset');+const store = await KeyValueStore.open('my-store');+const queue = await RequestQueue.open('my-queue');+```++You can also use the object form, which additionally allows opening a storage by ID:++```typescript+const dataset = await Dataset.open({ name: 'my-dataset' });++// Opening by ID (e.g. on the Apify platform):+const dataset = await Dataset.open({ id: 'WkzbQMuFYuamGv3YF' });+```++Opening the default storage (no arguments or `null`) still works as before:++```typescript+const dataset = await Dataset.open();+```++The same change applies to `CrawlingContext.getKeyValueStore()` and `CrawlingContext.pushData()` — both now accept `string | StorageIdentifier` for identifying the target storage.+## `transformRequestFunction` precedence in `enqueueLinks`The `transformRequestFunction` callback in `enqueueLinks` now runs **after** URL pattern filtering (`globs`, `regexps`, `pseudoUrls`) instead of before. This means it has the highest priority and can overwrite any request options set by patterns or the global `label` option.diff --git a/packages/basic-crawler/src/internals/basic-crawler.ts b/packages/basic-crawler/src/internals/basic-crawler.tsindex 2367cf80b..a584ee1c1 100644--- a/packages/basic-crawler/src/internals/basic-crawler.ts+++ b/packages/basic-crawler/src/internals/basic-crawler.ts@@ -26,6 +26,7 @@ import type {Source,StatisticsOptions,StatisticState,+ StorageIdentifier,} from '@crawlee/core';import {AutoscaledPool,@@ -1096,7 +1097,7 @@ export class BasicCrawler<log: this.log,pushData: this.pushData.bind(this),useState: this.useState.bind(this),- getKeyValueStore: async (idOrName?: string) => KeyValueStore.open(idOrName),+ getKeyValueStore: async (identifier?: string | StorageIdentifier) => KeyValueStore.open(identifier),registerDeferredCleanup: (cleanup: () => Promise<unknown>) => {deferredCleanup.push(cleanup);},@@ -1657,16 +1658,21 @@ export class BasicCrawler</*** Pushes data to the specified {@apilink Dataset}, or the default crawler {@apilink Dataset} by calling {@apilink Dataset.pushData}.*/- async pushData(data: Parameters<Dataset['pushData']>[0], datasetIdOrName?: string): Promise<void> {- const dataset = await this.getDataset(datasetIdOrName);+ async pushData(+ data: Parameters<Dataset['pushData']>[0],+ datasetIdentifier?: string | StorageIdentifier,+ ): Promise<void> {+ const dataset = await this.getDataset(datasetIdentifier);return dataset.pushData(data);}/*** Retrieves the specified {@apilink Dataset}, or the default crawler {@apilink Dataset}.*/- async getDataset(idOrName?: string): Promise<Dataset> {- return Dataset.open(idOrName, { config: serviceLocator.getConfiguration() });+ async getDataset(identifier?: string | StorageIdentifier): Promise<Dataset> {+ return Dataset.open(identifier, {+ config: serviceLocator.getConfiguration(),+ });}/**diff --git a/packages/core/src/crawlers/crawler_commons.ts b/packages/core/src/crawlers/crawler_commons.tsindex aec214e59..25de68e03 100644--- a/packages/core/src/crawlers/crawler_commons.ts+++ b/packages/core/src/crawlers/crawler_commons.ts@@ -9,6 +9,7 @@ import type { Session } from '../session_pool/session.js';import type { Dataset } from '../storages/dataset.js';import { KeyValueStore, type RecordOptions } from '../storages/key_value_store.js';import type { RequestQueueOperationOptions } from '../storages/request_provider.js';+import type { StorageIdentifier } from '../storages/storage_manager.js';/** @internal */export type IsAny<T> = 0 extends 1 & T ? true : false;@@ -48,7 +49,10 @@ export interface RestrictedCrawlingContext<UserData extends Dictionary = Diction** @param [data] Data to be pushed to the default dataset.*/- pushData(data: ReadonlyDeep<Parameters<Dataset['pushData']>[0]>, datasetIdOrName?: string): Promise<void>;+ pushData(+ data: ReadonlyDeep<Parameters<Dataset['pushData']>[0]>,+ datasetIdentifier?: string | StorageIdentifier,+ ): Promise<void>;/*** This function automatically finds and enqueues links from the current page, adding them to the {@apilink RequestQueue}@@ -98,7 +102,7 @@ export interface RestrictedCrawlingContext<UserData extends Dictionary = Diction* Get a key-value store with given name or id, or the default one for the crawler.*/getKeyValueStore: (- idOrName?: string,+ identifier?: string | StorageIdentifier,) => Promise<Pick<KeyValueStore, 'id' | 'name' | 'getValue' | 'getAutoSavedValue' | 'setValue' | 'getPublicUrl'>>;/**@@ -208,9 +212,9 @@ export class RequestHandlerResult {/*** Items added to datasets by a request handler.*/- get datasetItems(): ReadonlyDeep<{ item: Dictionary; datasetIdOrName?: string }[]> {- return this.pushDataCalls.flatMap(([data, datasetIdOrName]) =>- (Array.isArray(data) ? data : [data]).map((item) => ({ item, datasetIdOrName })),+ get datasetItems(): ReadonlyDeep<{ item: Dictionary; datasetIdentifier?: string | StorageIdentifier }[]> {+ return this.pushDataCalls.flatMap(([data, datasetIdentifier]) =>+ (Array.isArray(data) ? data : [data]).map((item) => ({ item, datasetIdentifier })),);}@@ -271,36 +275,35 @@ export class RequestHandlerResult {return await store.getAutoSavedValue(this.crawleeStateKey, defaultValue);};- getKeyValueStore: RestrictedCrawlingContext['getKeyValueStore'] = async (idOrName) => {- const store = await KeyValueStore.open(idOrName, { config: this.config });+ getKeyValueStore: RestrictedCrawlingContext['getKeyValueStore'] = async (identifier) => {+ const store = await KeyValueStore.open(identifier, { config: this.config });+ const storeId = store.id;return {- id: this.idOrDefault(idOrName),- name: idOrName,- getValue: async (key) => this.getKeyValueStoreChangedValue(idOrName, key) ?? (await store.getValue(key)),+ id: storeId ?? this.config.get('defaultKeyValueStoreId'),+ name: store.name,+ getValue: async (key) => this.getKeyValueStoreChangedValue(storeId, key) ?? (await store.getValue(key)),setValue: async (key, value, options) => {- this.setKeyValueStoreChangedValue(idOrName, key, value, options);+ this.setKeyValueStoreChangedValue(storeId, key, value, options);},getAutoSavedValue: store.getAutoSavedValue.bind(store),getPublicUrl: store.getPublicUrl.bind(store),};};- private idOrDefault = (idOrName?: string): string => idOrName ?? this.config.get('defaultKeyValueStoreId');-- private getKeyValueStoreChangedValue = (idOrName: string | undefined, key: string) => {- const id = this.idOrDefault(idOrName);+ private getKeyValueStoreChangedValue = (storeKey: string | undefined, key: string) => {+ const id = storeKey ?? this.config.get('defaultKeyValueStoreId') ?? 'default';this._keyValueStoreChanges[id] ??= {};return this.keyValueStoreChanges[id][key]?.changedValue ?? null;};private setKeyValueStoreChangedValue = (- idOrName: string | undefined,+ storeKey: string | undefined,key: string,changedValue: unknown,options?: RecordOptions,) => {- const id = this.idOrDefault(idOrName);+ const id = storeKey ?? this.config.get('defaultKeyValueStoreId') ?? 'default';this._keyValueStoreChanges[id] ??= {};this._keyValueStoreChanges[id][key] = { changedValue, options };};diff --git a/packages/core/src/recoverable_state.ts b/packages/core/src/recoverable_state.tsindex 76e28eb8a..f85ecad43 100644--- a/packages/core/src/recoverable_state.ts+++ b/packages/core/src/recoverable_state.ts@@ -119,9 +119,15 @@ export class RecoverableState<TStateModel = Record<string, unknown>> {return this.currentValue;}- this.keyValueStore = await KeyValueStore.open(this.persistStateKvsName ?? this.persistStateKvsId, {- config: serviceLocator.getConfiguration(),- });+ let kvsIdentifier: { name: string } | { id: string } | null = null;++ if (this.persistStateKvsName) {+ kvsIdentifier = { name: this.persistStateKvsName };+ } else if (this.persistStateKvsId) {+ kvsIdentifier = { id: this.persistStateKvsId };+ }++ this.keyValueStore = await KeyValueStore.open(kvsIdentifier, { config: serviceLocator.getConfiguration() });await this.loadSavedState();diff --git a/packages/core/src/session_pool/session_pool.ts b/packages/core/src/session_pool/session_pool.tsindex eda376c6e..0e040bf1d 100644--- a/packages/core/src/session_pool/session_pool.ts+++ b/packages/core/src/session_pool/session_pool.ts@@ -224,9 +224,12 @@ export class SessionPool extends EventEmitter {return;}- this.keyValueStore = await KeyValueStore.open(this.persistStateKeyValueStoreId, {- config: serviceLocator.getConfiguration(),- });+ this.keyValueStore = await KeyValueStore.open(+ this.persistStateKeyValueStoreId ? { id: this.persistStateKeyValueStoreId } : null,+ {+ config: serviceLocator.getConfiguration(),+ },+ );if (!this.persistStateKeyValueStoreId) {this.log.debug(diff --git a/packages/core/src/storages/dataset.ts b/packages/core/src/storages/dataset.tsindex dd4e3dec3..f2a903912 100644--- a/packages/core/src/storages/dataset.ts+++ b/packages/core/src/storages/dataset.ts@@ -1,4 +1,4 @@-import type { DatasetClient, DatasetInfo, Dictionary, PaginatedList, StorageClient } from '@crawlee/types';+import type { DatasetClient, DatasetInfo, Dictionary, PaginatedList } from '@crawlee/types';import { stringify } from 'csv-stringify/sync';import ow from 'ow';@@ -10,7 +10,7 @@ import { serviceLocator } from '../service_locator.js';import type { Awaitable } from '../typedefs.js';import { checkStorageAccess } from './access_checking.js';import { KeyValueStore } from './key_value_store.js';-import type { StorageManagerOptions } from './storage_manager.js';+import type { StorageIdentifier, StorageManagerOptions } from './storage_manager.js';import { StorageManager } from './storage_manager.js';import { purgeDefaultStorages } from './utils.js';@@ -177,8 +177,8 @@ export interface DatasetIteratorOptions extends Omit<}export interface DatasetExportToOptions extends DatasetExportOptions {- fromDataset?: string;- toKVS?: string;+ fromDataset?: string | StorageIdentifier;+ toKVS?: string | StorageIdentifier;}/**@@ -236,7 +236,6 @@ export class Dataset<Data extends Dictionary = Dictionary> {id: string;name?: string;client: DatasetClient<Data>;- readonly storageObject?: Record<string, unknown>;log: CrawleeLogger;/**@@ -248,8 +247,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {) {this.id = options.id;this.name = options.name;- this.client = options.client.dataset(this.id) as DatasetClient<Data>;- this.storageObject = options.storageObject;+ this.client = options.client;this.log = serviceLocator.getLogger().child({ prefix: 'Dataset' });}@@ -438,29 +436,24 @@ export class Dataset<Data extends Dictionary = Dictionary> {/*** Returns an object containing general information about the dataset.*- * The function returns the same object as the Apify API Client's- * [DatasetClient.get()](https://docs.apify.com/api/client/js/reference/class/DatasetClient#get)- * function, which in turn calls the- * [Get dataset](https://apify.com/docs/api/v2#/reference/datasets/dataset/get-dataset)- * API endpoint.- ** **Example:*** ```* {* id: "WkzbQMuFYuamGv3YF",* name: "my-dataset",- * userId: "wRsJZtadYvn4mBZmm",* createdAt: new Date("2015-12-12T07:34:14.202Z"),* modifiedAt: new Date("2015-12-13T08:36:13.202Z"),* accessedAt: new Date("2015-12-14T08:36:13.202Z"),* itemCount: 14,* }* ```+ *+ * @throws If the underlying storage no longer exists (e.g. it was deleted externally).*/- async getInfo(): Promise<DatasetInfo | undefined> {+ async getInfo(): Promise<DatasetInfo> {checkStorageAccess();- return this.client.get();+ return this.client.getMetadata();}/**@@ -701,18 +694,18 @@ export class Dataset<Data extends Dictionary = Dictionary> {** For more details and code examples, see the {@apilink Dataset} class.*- * @param [datasetIdOrName]- * ID or name of the dataset to be opened. If `null` or `undefined`,- * the function returns the default dataset associated with the crawler run.+ * @param [identifier]+ * ID or name of the dataset to be opened. If a string is provided, it will first be+ * looked up as an ID; if no such storage exists, it will be treated as a name.+ * If `null` or `undefined`, the function returns the default dataset associated with the crawler run.* @param [options] Storage manager options.*/static async open<Data extends Dictionary = Dictionary>(- datasetIdOrName?: string | null,+ identifier?: string | StorageIdentifier | null,options: StorageManagerOptions = {},): Promise<Dataset<Data>> {checkStorageAccess();- ow(datasetIdOrName, ow.optional.string);ow(options,ow.object.exactShape({@@ -728,7 +721,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {const manager = StorageManager.getManager<Dataset<Data>>(this);- return manager.openStorage(datasetIdOrName, options.storageClient);+ return manager.openStorage(identifier, options.storageClient);}/**@@ -809,8 +802,7 @@ export interface DatasetReducer<T, Data> {export interface DatasetOptions {id: string;name?: string;- client: StorageClient;- storageObject?: Record<string, unknown>;+ client: DatasetClient;}export interface DatasetContent<Data> {diff --git a/packages/core/src/storages/key_value_store.ts b/packages/core/src/storages/key_value_store.tsindex 72346ebdf..732474874 100644--- a/packages/core/src/storages/key_value_store.ts+++ b/packages/core/src/storages/key_value_store.ts@@ -1,7 +1,7 @@import { readFile } from 'node:fs/promises';import { join } from 'node:path';-import type { Dictionary, KeyValueStoreClient, StorageClient } from '@crawlee/types';+import type { Dictionary, KeyValueStoreClient } from '@crawlee/types';import JSON5 from 'json5';import ow, { ArgumentError } from 'ow';@@ -12,7 +12,7 @@ import { Configuration } from '../configuration.js';import { serviceLocator } from '../service_locator.js';import type { Awaitable } from '../typedefs.js';import { checkStorageAccess } from './access_checking.js';-import type { StorageManagerOptions } from './storage_manager.js';+import type { StorageIdentifier, StorageManagerOptions } from './storage_manager.js';import { StorageManager } from './storage_manager.js';import { purgeDefaultStorages } from './utils.js';@@ -108,7 +108,6 @@ export const maybeStringify = <T>(value: T, options: { contentType?: string }) =export class KeyValueStore {readonly id: string;readonly name?: string;- readonly storageObject?: Record<string, unknown>;private readonly client: KeyValueStoreClient;private persistStateEventStarted = false;@@ -124,8 +123,7 @@ export class KeyValueStore {) {this.id = options.id;this.name = options.name;- this.storageObject = options.storageObject;- this.client = options.client.keyValueStore(this.id);+ this.client = options.client;}/**@@ -593,15 +591,18 @@ export class KeyValueStore {** For more details and code examples, see the {@apilink KeyValueStore} class.*- * @param [storeIdOrName]- * ID or name of the key-value store to be opened. If `null` or `undefined`,- * the function returns the default key-value store associated with the crawler run.+ * @param [identifier]+ * ID or name of the key-value store to be opened. If a string is provided, it will first be+ * looked up as an ID; if no such storage exists, it will be treated as a name.+ * If `null` or `undefined`, the function returns the default key-value store associated with the crawler run.* @param [options] Storage manager options.*/- static async open(storeIdOrName?: string | null, options: StorageManagerOptions = {}): Promise<KeyValueStore> {+ static async open(+ identifier?: string | StorageIdentifier | null,+ options: StorageManagerOptions = {},+ ): Promise<KeyValueStore> {checkStorageAccess();- ow(storeIdOrName, ow.optional.any(ow.string, ow.null));ow(options,ow.object.exactShape({@@ -617,7 +618,7 @@ export class KeyValueStore {const manager = StorageManager.getManager(this);- return manager.openStorage(storeIdOrName, options.storageClient);+ return manager.openStorage(identifier, options.storageClient);}/**@@ -829,8 +830,7 @@ export interface KeyConsumer {export interface KeyValueStoreOptions {id: string;name?: string;- client: StorageClient;- storageObject?: Record<string, unknown>;+ client: KeyValueStoreClient;}export interface RecordOptions {diff --git a/packages/core/src/storages/request_provider.ts b/packages/core/src/storages/request_provider.tsindex ff36a5040..d2618bf6d 100644--- a/packages/core/src/storages/request_provider.ts+++ b/packages/core/src/storages/request_provider.ts@@ -8,7 +8,6 @@ import type {QueueOperationInfo,RequestQueueClient,RequestQueueInfo,- StorageClient,} from '@crawlee/types';import {chunkedAsyncIterable,@@ -36,7 +35,7 @@ import { serviceLocator } from '../service_locator.js';import type { Constructor } from '../typedefs.js';import { checkStorageAccess } from './access_checking.js';import { RequestDeduplicationCache } from './request_dedup_cache.js';-import type { IStorage, StorageManagerOptions } from './storage_manager.js';+import type { IStorage, StorageIdentifier, StorageManagerOptions } from './storage_manager.js';import { StorageManager } from './storage_manager.js';import { getRequestId, purgeDefaultStorages, QUERY_HEAD_MIN_LENGTH } from './utils.js';@@ -160,10 +159,7 @@ export abstract class RequestProvider implements IStorage, IRequestManager {this.id = options.id;this.name = options.name;this.events = serviceLocator.getEventManager();- this.client = options.client.requestQueue(this.id, {- clientKey: this.clientKey,- timeoutSecs: this.timeoutSecs,- });+ this.client = options.client;this.proxyConfiguration = options.proxyConfiguration;@@ -835,25 +831,18 @@ export abstract class RequestProvider implements IStorage, IRequestManager {*/async handledCount(): Promise<number> {// NOTE: We keep this function for compatibility with RequestList.handledCount()- const { handledRequestCount } = (await this.getInfo()) ?? {};- return handledRequestCount ?? 0;+ const { handledRequestCount } = await this.getInfo();+ return handledRequestCount;}/*** Returns an object containing general information about the request queue.*- * The function returns the same object as the Apify API Client's- * [RequestQueueClient.get()](https://docs.apify.com/api/client/js/reference/class/RequestQueueClient#get)- * function, which in turn calls the- * [Get request queue](https://apify.com/docs/api/v2#/reference/request-queues/queue/get-request-queue)- * API endpoint.- ** **Example:*** ```* {* id: "WkzbQMuFYuamGv3YF",* name: "my-queue",- * userId: "wRsJZtadYvn4mBZmm",* createdAt: new Date("2015-12-12T07:34:14.202Z"),* modifiedAt: new Date("2015-12-13T08:36:13.202Z"),* accessedAt: new Date("2015-12-14T08:36:13.202Z"),@@ -862,11 +851,13 @@ export abstract class RequestProvider implements IStorage, IRequestManager {* pendingRequestCount: 20,* }* ```+ *+ * @throws If the underlying storage no longer exists (e.g. it was deleted externally).*/- async getInfo(): Promise<RequestQueueInfo | undefined> {+ async getInfo(): Promise<RequestQueueInfo> {checkStorageAccess();- return this.client.get();+ return this.client.getMetadata();}/**@@ -944,15 +935,18 @@ export abstract class RequestProvider implements IStorage, IRequestManager {** For more details and code examples, see the {@apilink RequestQueue} class.*- * @param [queueIdOrName]- * ID or name of the request queue to be opened. If `null` or `undefined`,- * the function returns the default request queue associated with the crawler run.+ * @param [identifier]+ * ID or name of the request queue to be opened. If a string is provided, it will first be+ * looked up as an ID; if no such storage exists, it will be treated as a name.+ * If `null` or `undefined`, the function returns the default request queue associated with the crawler run.* @param [options] Open Request Queue options.*/- static async open(queueIdOrName?: string | null, options: StorageManagerOptions = {}): Promise<RequestProvider> {+ static async open(+ identifier?: string | StorageIdentifier | null,+ options: StorageManagerOptions = {},+ ): Promise<RequestProvider> {checkStorageAccess();- ow(queueIdOrName, ow.optional.any(ow.string, ow.null));ow(options,ow.object.exactShape({@@ -968,13 +962,24 @@ export abstract class RequestProvider implements IStorage, IRequestManager {await purgeDefaultStorages({ onlyPurgeOnce: true, client: options.storageClient, config: options.config });const manager = StorageManager.getManager(this as typeof BuiltRequestProvider);- const queue = await manager.openStorage(queueIdOrName, options.storageClient);+ const queue = await manager.openStorage(identifier, options.storageClient);queue.proxyConfiguration = options.proxyConfiguration;- const queueInfo = await queue.client.get();+ // Re-create the request queue client with clientKey and timeoutSecs so that+ // request locking works correctly for API-backed implementations.+ // TODO: clientKey/timeoutSecs are Apify-platform concerns and should eventually be pushed+ // down into the Apify SDK's client implementation, aligning with crawlee-python's approach+ // where locking is handled internally by the client (see crawlee-python PR #1194).+ queue.client = await options.storageClient.createRequestQueueClient({+ id: queue.id,+ clientKey: queue.clientKey,+ timeoutSecs: queue.timeoutSecs,+ });++ const queueInfo = await queue.client.getMetadata();- queue.initialCount = queueInfo?.totalRequestCount ?? 0;- queue.initialHandledCount = queueInfo?.handledRequestCount ?? 0;+ queue.initialCount = queueInfo.totalRequestCount;+ queue.initialHandledCount = queueInfo.handledRequestCount;queue.httpClient = options.httpClient;return queue;@@ -1003,7 +1008,7 @@ interface RequestLruItem {export interface RequestProviderOptions {id: string;name?: string;- client: StorageClient;+ client: RequestQueueClient;/*** Used to pass the proxy configuration for the `requestsFromUrl` objects.diff --git a/packages/core/src/storages/storage_manager.ts b/packages/core/src/storages/storage_manager.tsindex 634132a32..3908d7b44 100644--- a/packages/core/src/storages/storage_manager.ts+++ b/packages/core/src/storages/storage_manager.ts@@ -1,4 +1,11 @@-import type { BaseHttpClient, StorageClient } from '@crawlee/types';+import type {+ BaseHttpClient,+ DatasetClient,+ KeyValueStoreClient,+ RequestQueueClient,+ StorageClient,+ StorageIdentifier,+} from '@crawlee/types';import { AsyncQueue } from '@sapphire/async-queue';import type { Configuration } from '../configuration.js';@@ -6,6 +13,8 @@ import type { ProxyConfiguration } from '../proxy_configuration.js';import { serviceLocator } from '../service_locator.js';import type { Constructor } from '../typedefs.js';+export type { StorageIdentifier } from '@crawlee/types';+const DEFAULT_ID_CONFIG_KEYS = {Dataset: 'defaultDatasetId',KeyValueStore: 'defaultKeyValueStoreId',@@ -37,10 +46,10 @@ export class StorageManager<T extends IStorage = IStorage> {static async openStorage<T extends IStorage>(storageClass: Constructor<T>,- idOrName?: string,+ identifier?: string | StorageIdentifier,client?: StorageClient,): Promise<T> {- return this.getManager(storageClass).openStorage(idOrName, client);+ return this.getManager(storageClass).openStorage(identifier, client);}static getManager<T extends IStorage>(storageClass: Constructor<T>): StorageManager<T> {@@ -53,27 +62,26 @@ export class StorageManager<T extends IStorage = IStorage> {return serviceLocator.getStorageManager(storageClass) as StorageManager<T>;}- async openStorage(idOrName?: string | null, client?: StorageClient): Promise<T> {+ async openStorage(identifier?: string | StorageIdentifier | null, client?: StorageClient): Promise<T> {await this.storageOpenQueue.wait();try {- if (!idOrName) {- const defaultIdConfigKey = DEFAULT_ID_CONFIG_KEYS[this.name];- idOrName = this.config.get(defaultIdConfigKey) as string;- }+ const resolvedIdentifier = await this._resolveIdentifier(identifier, client);- const cacheKey = idOrName;+ const cacheKey = resolvedIdentifier.id ?? resolvedIdentifier.name!;let storage = this.cache.get(cacheKey);if (!storage) {client ??= serviceLocator.getStorageClient();- const storageObject = await this._getOrCreateStorage(idOrName, this.name, client);+ const subClient = await this._createSubClient(resolvedIdentifier, this.name, client);+ const storageInfo = await (+ subClient as DatasetClient | KeyValueStoreClient | RequestQueueClient+ ).getMetadata();storage = new this.StorageConstructor({- id: storageObject.id,- name: storageObject.name,- storageObject,- client,+ id: storageInfo.id,+ name: storageInfo.name,+ client: subClient,},this.config,);@@ -86,6 +94,42 @@ export class StorageManager<T extends IStorage = IStorage> {}}+ /**+ * Resolves the user-provided identifier to an unambiguous `StorageIdentifier`.+ *+ * - `null`/`undefined` → uses the default storage ID from config+ * - `StorageIdentifier` object → passed through (with default ID fallback if empty)+ * - `string` → tries to find an existing storage with that ID first;+ * if none exists, treats the string as a name+ */+ private async _resolveIdentifier(+ identifier?: string | StorageIdentifier | null,+ client?: StorageClient,+ ): Promise<StorageIdentifier> {+ if (typeof identifier === 'string') {+ client ??= serviceLocator.getStorageClient();++ if (client.storageExists && (await client.storageExists(identifier, this.name))) {+ return { id: identifier };+ }++ return { name: identifier };+ }++ if (identifier?.id) {+ return { id: identifier.id };+ }++ if (identifier?.name) {+ return { name: identifier.name };+ }++ const defaultIdConfigKey = DEFAULT_ID_CONFIG_KEYS[this.name];+ const defaultId = this.config.get(defaultIdConfigKey) as string;++ return { id: defaultId };+ }+closeStorage(storage: { id: string; name?: string }): void {const idKey = storage.id;this.cache.delete(idKey);@@ -97,36 +141,23 @@ export class StorageManager<T extends IStorage = IStorage> {}/**- * Helper function that first requests storage by ID and if storage doesn't exist then gets it by name.+ * Creates a sub-client for the given storage type using the new factory-based StorageClient interface.*/- protected async _getOrCreateStorage(- storageIdOrName: string,+ protected async _createSubClient(+ identifier: StorageIdentifier,storageConstructorName: string,apiClient: StorageClient,- ) {- const { createStorageClient, createStorageCollectionClient } = this._getStorageClientFactories(- apiClient,- storageConstructorName,- );-- const storageClient = createStorageClient(storageIdOrName);- const existingStorage = await storageClient.get();- if (existingStorage) return existingStorage;-- const storageCollectionClient = createStorageCollectionClient();- return storageCollectionClient.getOrCreate(storageIdOrName);- }-- protected _getStorageClientFactories(client: StorageClient, storageConstructorName: string) {- // Dataset => dataset- const clientName = (storageConstructorName[0].toLowerCase() + storageConstructorName.slice(1)) as ClientNames;- // dataset => datasets- const collectionClientName = `${clientName}s` as ClientCollectionNames;-- return {- createStorageClient: client[clientName!].bind(client),- createStorageCollectionClient: client[collectionClientName!].bind(client),- };+ ): Promise<DatasetClient | KeyValueStoreClient | RequestQueueClient> {+ switch (storageConstructorName) {+ case 'Dataset':+ return apiClient.createDatasetClient(identifier);+ case 'KeyValueStore':+ return apiClient.createKeyValueStoreClient(identifier);+ case 'RequestQueue':+ return apiClient.createRequestQueueClient(identifier);+ default:+ throw new Error(`Unknown storage type: ${storageConstructorName}`);+ }}protected _addStorageToCache(storage: T): void {@@ -140,9 +171,6 @@ export class StorageManager<T extends IStorage = IStorage> {}}-type ClientNames = 'dataset' | 'keyValueStore' | 'requestQueue';-type ClientCollectionNames = 'datasets' | 'keyValueStores' | 'requestQueues';-export interface StorageManagerOptions {/*** SDK configuration instance, defaults to the static register.diff --git a/packages/core/src/storages/utils.ts b/packages/core/src/storages/utils.tsindex ec0881162..5a8788b9e 100644--- a/packages/core/src/storages/utils.ts+++ b/packages/core/src/storages/utils.ts@@ -100,7 +100,7 @@ export async function useState<State extends Dictionary = Dictionary>(defaultValue = {} as State,options?: UseStateOptions,) {- const kvStore = await KeyValueStore.open(options?.keyValueStoreName, {+ const kvStore = await KeyValueStore.open(options?.keyValueStoreName ? { name: options.keyValueStoreName } : null, {config: options?.config || serviceLocator.getConfiguration(),});return kvStore.getAutoSavedValue<State>(name || 'CRAWLEE_GLOBAL_STATE', defaultValue);diff --git a/packages/core/test/request-queue/adding-the-same-request-should-not-call-the-api.test.ts b/packages/core/test/request-queue/adding-the-same-request-should-not-call-the-api.test.tsindex cd212bbc5..bcc7ebc16 100644--- a/packages/core/test/request-queue/adding-the-same-request-should-not-call-the-api.test.ts+++ b/packages/core/test/request-queue/adding-the-same-request-should-not-call-the-api.test.ts@@ -1,22 +1,20 @@import { MemoryStorage } from '@crawlee/memory-storage';-import type { RequestQueueInfo } from '@crawlee/types';+import type { RequestQueueClient } from '@crawlee/types';import { RequestQueue, serviceLocator } from 'crawlee';-let requestQueueInfo: RequestQueueInfo;+let rqClient: RequestQueueClient;beforeEach(async () => {- serviceLocator.setStorageClient(new MemoryStorage({ persistStorage: false, writeMetadata: false }));- requestQueueInfo = await serviceLocator- .getStorageClient()- .requestQueues()- .getOrCreate('test-request-queue-not-called-on-cached-request');+ const storage = new MemoryStorage({ persistStorage: false, writeMetadata: false });+ serviceLocator.setStorageClient(storage);+ rqClient = await storage.createRequestQueueClient({ name: 'test-request-queue-not-called-on-cached-request' });});describe('RequestQueue#addRequest should not call the API if the request is already in the queue', () => {test('should not call the API if the request is already in the queue', async () => {- const client = serviceLocator.getStorageClient();const config = serviceLocator.getConfiguration();- const requestQueue = new RequestQueue({ id: requestQueueInfo.id, client }, config);+ const rqInfo = await rqClient.getMetadata();+ const requestQueue = new RequestQueue({ id: rqInfo.id, client: rqClient }, config);const clientSpy = vitest.spyOn(requestQueue.client, 'addRequest');@@ -38,9 +36,9 @@ describe('RequestQueue#addRequest should not call the API if the request is alredescribe('RequestQueue#addRequests should not call the API if the request is already in the queue', () => {test('should not call the API if the request is already in the queue', async () => {- const client = serviceLocator.getStorageClient();const config = serviceLocator.getConfiguration();- const requestQueue = new RequestQueue({ id: requestQueueInfo.id, client }, config);+ const rqInfo = await rqClient.getMetadata();+ const requestQueue = new RequestQueue({ id: rqInfo.id, client: rqClient }, config);const clientSpy = vitest.spyOn(requestQueue.client, 'batchAddRequests');diff --git a/packages/core/test/request-queue/request-queue-v2.test.ts b/packages/core/test/request-queue/request-queue-v2.test.tsindex d3b0affdd..802c0ca79 100644--- a/packages/core/test/request-queue/request-queue-v2.test.ts+++ b/packages/core/test/request-queue/request-queue-v2.test.ts@@ -1,14 +1,15 @@import { MemoryStorage } from '@crawlee/memory-storage';-import type { ListAndLockHeadResult } from '@crawlee/types';+import type { ListAndLockHeadResult, RequestQueueClient } from '@crawlee/types';import { RequestQueueV2 } from 'crawlee';import type { MockInstance } from 'vitest';const storage = new MemoryStorage({ persistStorage: false, writeMetadata: false });async function makeQueue(name: string, numOfRequestsToAdd = 0) {- const queueData = await storage.requestQueues().getOrCreate(name);+ const rqClient = await storage.createRequestQueueClient({ name });+ const rqInfo = await rqClient.getMetadata();- const queue = new RequestQueueV2({ id: queueData.id, client: storage });+ const queue = new RequestQueueV2({ id: rqInfo.id, client: rqClient });if (numOfRequestsToAdd) {await queue.addRequests(diff --git a/packages/core/test/storages/open-storage-with-different-client-should-be-respected.test.ts b/packages/core/test/storages/open-storage-with-different-client-should-be-respected.test.tsindex 48b15ed25..3c479394c 100644--- a/packages/core/test/storages/open-storage-with-different-client-should-be-respected.test.ts+++ b/packages/core/test/storages/open-storage-with-different-client-should-be-respected.test.ts@@ -10,8 +10,10 @@ beforeEach(() => {describe('Opening a storage with a different storage client should be respected', () => {test('opening a RequestQueue with default client from Configuration', async () => {- const queue = await RequestQueue.open('test-rq-open-client-from-config');+ const queue = await RequestQueue.open({ name: 'test-rq-open-client-from-config' });+ // The sub-client should have been created by newClient (MemoryStorage),+ // so its internal `client` field should reference newClient.expect((queue.client as any).client).toBe(newClient);});@@ -20,7 +22,7 @@ describe('Opening a storage with a different storage client should be respected'// @ts-expect-error Using this to ensure the test/impl worksthirdClient._name = 'third-client';- const queue = await RequestQueue.open('test-rq-open-custom-client', { storageClient: thirdClient });+ const queue = await RequestQueue.open({ name: 'test-rq-open-custom-client' }, { storageClient: thirdClient });expect((queue.client as any).client).toBe(thirdClient);});diff --git a/packages/http-crawler/src/internals/http-crawler.ts b/packages/http-crawler/src/internals/http-crawler.tsindex 11552bf53..3378efad7 100644--- a/packages/http-crawler/src/internals/http-crawler.ts+++ b/packages/http-crawler/src/internals/http-crawler.ts@@ -671,7 +671,9 @@ export class HttpCrawler<// Try to re-encode a variety of unsupported encodings to utf-8if (iconv.encodingExists(encoding)) {const encodeStream = iconv.encodeStream(utf8);- const decodeStream = iconv.decodeStream(encoding).on('error', (err) => encodeStream.emit('error', err));+ const decodeStream = iconv+ .decodeStream(encoding)+ .on('error', (err: Error) => encodeStream.emit('error', err));const reencodedBody = response.body? Readable.toWeb(Readable.from(diff --git a/packages/memory-storage/src/cache-helpers.ts b/packages/memory-storage/src/cache-helpers.tsindex 69ab0001f..3d213e1ea 100644--- a/packages/memory-storage/src/cache-helpers.ts+++ b/packages/memory-storage/src/cache-helpers.ts@@ -14,7 +14,7 @@ const uuidRegex = /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/export async function findOrCacheDatasetByPossibleId(client: MemoryStorage, entryNameOrId: string) {// First check memory cache- const found = client.datasetClientsHandled.find(+ const found = client.datasetClientCache.find((store) => store.id === entryNameOrId || store.name?.toLowerCase() === entryNameOrId.toLowerCase(),);@@ -110,14 +110,14 @@ export async function findOrCacheDatasetByPossibleId(client: MemoryStorage, entrnewClient['datasetEntries'].set(entryId, entry);}- client.datasetClientsHandled.push(newClient);+ client.datasetClientCache.push(newClient);return newClient;}export async function findOrCacheKeyValueStoreByPossibleId(client: MemoryStorage, entryNameOrId: string) {// First check memory cache- const found = client.keyValueStoresHandled.find(+ const found = client.keyValueStoreCache.find((store) => store.id === entryNameOrId || store.name?.toLowerCase() === entryNameOrId.toLowerCase(),);@@ -264,14 +264,14 @@ export async function findOrCacheKeyValueStoreByPossibleId(client: MemoryStoragenewClient['keyValueEntries'].set(key, entry);}- client.keyValueStoresHandled.push(newClient);+ client.keyValueStoreCache.push(newClient);return newClient;}export async function findRequestQueueByPossibleId(client: MemoryStorage, entryNameOrId: string) {// First check memory cache- const found = client.requestQueuesHandled.find(+ const found = client.requestQueueCache.find((store) => store.id === entryNameOrId || store.name?.toLowerCase() === entryNameOrId.toLowerCase(),);@@ -383,7 +383,7 @@ export async function findRequestQueueByPossibleId(client: MemoryStorage, entryNnewClient['requests'].set(requestId, entry);}- client.requestQueuesHandled.push(newClient);+ client.requestQueueCache.push(newClient);return newClient;}diff --git a/packages/memory-storage/src/memory-storage.ts b/packages/memory-storage/src/memory-storage.tsindex ec50f24bf..a5b33fb63 100644--- a/packages/memory-storage/src/memory-storage.ts+++ b/packages/memory-storage/src/memory-storage.ts@@ -1,20 +1,21 @@/* eslint-disable import/no-duplicates */-import { readdir, rm } from 'node:fs/promises';+import { access, readdir, rm } from 'node:fs/promises';import { resolve } from 'node:path';import type * as storage from '@crawlee/types';-import type { Dictionary } from '@crawlee/types';import type { CrawleeLogger } from '@crawlee/types';import { s } from '@sapphire/shapeshift';import { ensureDirSync, move, moveSync, pathExistsSync } from 'fs-extra/esm';-import { promiseMap } from './background-handler/index.js';+import { promiseMap, scheduleBackgroundTask } from './background-handler/index.js';+import {+ findOrCacheDatasetByPossibleId,+ findOrCacheKeyValueStoreByPossibleId,+ findRequestQueueByPossibleId,+} from './cache-helpers.js';import { DatasetClient } from './resource-clients/dataset.js';-import { DatasetCollectionClient } from './resource-clients/dataset-collection.js';import { KeyValueStoreClient } from './resource-clients/key-value-store.js';-import { KeyValueStoreCollectionClient } from './resource-clients/key-value-store-collection.js';import { RequestQueueClient } from './resource-clients/request-queue.js';-import { RequestQueueCollectionClient } from './resource-clients/request-queue-collection.js';export interface MemoryStorageOptions {/**@@ -52,9 +53,9 @@ export class MemoryStorage implements storage.StorageClient {readonly persistStorage: boolean;readonly logger?: CrawleeLogger;- readonly keyValueStoresHandled: KeyValueStoreClient[] = [];- readonly datasetClientsHandled: DatasetClient[] = [];- readonly requestQueuesHandled: RequestQueueClient[] = [];+ readonly keyValueStoreCache: KeyValueStoreClient[] = [];+ readonly datasetClientCache: DatasetClient[] = [];+ readonly requestQueueCache: RequestQueueClient[] = [];constructor(options: MemoryStorageOptions = {}) {s.object({@@ -92,52 +93,150 @@ export class MemoryStorage implements storage.StorageClient {: true);}- datasets(): storage.DatasetCollectionClient {- return new DatasetCollectionClient({- baseStorageDirectory: this.datasetsDirectory,- client: this,- });- }+ async createDatasetClient(options: storage.CreateDatasetClientOptions = {}): Promise<storage.DatasetClient> {+ // In MemoryStorage, both id and name resolve to the same directory name.+ const name = options.name ?? options.id;- dataset<Data extends Dictionary = Dictionary>(id: string): storage.DatasetClient<Data> {- s.string().parse(id);+ if (name) {+ const found = await findOrCacheDatasetByPossibleId(this, name);+ if (found) {+ return found;+ }+ }- return new DatasetClient({ id, baseStorageDirectory: this.datasetsDirectory, client: this });+ const newStore = new DatasetClient({ name, baseStorageDirectory: this.datasetsDirectory, client: this });+ this.datasetClientCache.push(newStore);++ // Schedule the worker to write to the disk+ const datasetInfo = newStore.toDatasetInfo();++ scheduleBackgroundTask(+ {+ action: 'update-metadata',+ entityType: 'datasets',+ entityDirectory: newStore.datasetDirectory,+ id: datasetInfo.name ?? datasetInfo.id,+ data: datasetInfo,+ writeMetadata: this.writeMetadata,+ persistStorage: this.persistStorage,+ },+ this.logger,+ );++ return newStore;}- keyValueStores(): storage.KeyValueStoreCollectionClient {- return new KeyValueStoreCollectionClient({+ async createKeyValueStoreClient(+ options: storage.CreateKeyValueStoreClientOptions = {},+ ): Promise<storage.KeyValueStoreClient> {+ // In MemoryStorage, both id and name resolve to the same directory name.+ const name = options.name ?? options.id;++ if (name) {+ const found = await findOrCacheKeyValueStoreByPossibleId(this, name);+ if (found) {+ return found;+ }+ }++ const newStore = new KeyValueStoreClient({+ name,baseStorageDirectory: this.keyValueStoresDirectory,client: this,});+ this.keyValueStoreCache.push(newStore);++ // Schedule the worker to write to the disk+ const kvStoreInfo = newStore.toKeyValueStoreInfo();++ scheduleBackgroundTask(+ {+ action: 'update-metadata',+ entityType: 'keyValueStores',+ entityDirectory: newStore.keyValueStoreDirectory,+ id: kvStoreInfo.name ?? kvStoreInfo.id,+ data: kvStoreInfo,+ writeMetadata: this.writeMetadata,+ persistStorage: this.persistStorage,+ },+ this.logger,+ );++ return newStore;}- keyValueStore(id: string): storage.KeyValueStoreClient {- s.string().parse(id);+ async createRequestQueueClient(+ options: storage.CreateRequestQueueClientOptions = {},+ ): Promise<storage.RequestQueueClient> {+ // In MemoryStorage, both id and name resolve to the same directory name.+ const name = options.name ?? options.id;- return new KeyValueStoreClient({ id, baseStorageDirectory: this.keyValueStoresDirectory, client: this });- }+ if (name) {+ const found = await findRequestQueueByPossibleId(this, name);+ if (found) {+ return found;+ }+ }- requestQueues(): storage.RequestQueueCollectionClient {- return new RequestQueueCollectionClient({+ const newStore = new RequestQueueClient({+ name,baseStorageDirectory: this.requestQueuesDirectory,client: this,});+ this.requestQueueCache.push(newStore);++ // Schedule the worker to write to the disk+ const queueInfo = newStore.toRequestQueueInfo();++ scheduleBackgroundTask(+ {+ action: 'update-metadata',+ entityType: 'requestQueues',+ entityDirectory: newStore.requestQueueDirectory,+ id: queueInfo.name ?? queueInfo.id,+ data: queueInfo,+ writeMetadata: this.writeMetadata,+ persistStorage: this.persistStorage,+ },+ this.logger,+ );++ return newStore;}- requestQueue(id: string, options: storage.RequestQueueOptions = {}): storage.RequestQueueClient {- s.string().parse(id);- s.object({- clientKey: s.string().optional(),- timeoutSecs: s.number().optional(),- }).parse(options);+ async storageExists(id: string, type: 'Dataset' | 'KeyValueStore' | 'RequestQueue'): Promise<boolean> {+ let clients: { id: string }[];+ let baseDir: string;++ switch (type) {+ case 'Dataset':+ clients = this.datasetClientCache;+ baseDir = this.datasetsDirectory;+ break;+ case 'KeyValueStore':+ clients = this.keyValueStoreCache;+ baseDir = this.keyValueStoresDirectory;+ break;+ case 'RequestQueue':+ clients = this.requestQueueCache;+ baseDir = this.requestQueuesDirectory;+ break;+ default:+ return false;+ }- return new RequestQueueClient({- id,- baseStorageDirectory: this.requestQueuesDirectory,- client: this,- ...options,- });+ // Check in-memory cache first+ if (clients.some((store) => store.id === id)) {+ return true;+ }++ // Check if a directory with that ID exists on disk+ try {+ await access(resolve(baseDir, id));+ return true;+ } catch {+ return false;+ }}async setStatusMessage(message: string, options: storage.SetStatusMessageOptions = {}): Promise<void> {diff --git a/packages/memory-storage/src/resource-clients/dataset-collection.ts b/packages/memory-storage/src/resource-clients/dataset-collection.tsdeleted file mode 100644index 104e9e55c..000000000--- a/packages/memory-storage/src/resource-clients/dataset-collection.ts+++ /dev/null@@ -1,70 +0,0 @@-import { resolve } from 'node:path';--import type * as storage from '@crawlee/types';-import { s } from '@sapphire/shapeshift';--import { scheduleBackgroundTask } from '../background-handler/index.js';-import { findOrCacheDatasetByPossibleId } from '../cache-helpers.js';-import type { MemoryStorage } from '../index.js';-import { DatasetClient } from './dataset.js';--export interface DatasetCollectionClientOptions {- baseStorageDirectory: string;- client: MemoryStorage;-}--export class DatasetCollectionClient implements storage.DatasetCollectionClient {- private readonly datasetsDirectory: string;- private readonly client: MemoryStorage;-- constructor({ baseStorageDirectory, client }: DatasetCollectionClientOptions) {- this.datasetsDirectory = resolve(baseStorageDirectory);- this.client = client;- }-- async list(): ReturnType<storage.DatasetCollectionClient['list']> {- return {- total: this.client.datasetClientsHandled.length,- count: this.client.datasetClientsHandled.length,- offset: 0,- limit: this.client.datasetClientsHandled.length,- desc: false,- items: this.client.datasetClientsHandled- .map((store) => store.toDatasetInfo())- .sort((a, b) => a.createdAt.getTime() - b.createdAt.getTime()),- };- }-- async getOrCreate(name?: string): Promise<storage.DatasetInfo> {- s.string().optional().parse(name);-- if (name) {- const found = await findOrCacheDatasetByPossibleId(this.client, name);-- if (found) {- return found.toDatasetInfo();- }- }-- const newStore = new DatasetClient({ name, baseStorageDirectory: this.datasetsDirectory, client: this.client });- this.client.datasetClientsHandled.push(newStore);-- // Schedule the worker to write to the disk- const datasetInfo = newStore.toDatasetInfo();-- scheduleBackgroundTask(- {- action: 'update-metadata',- entityType: 'datasets',- entityDirectory: newStore.datasetDirectory,- id: datasetInfo.name ?? datasetInfo.id,- data: datasetInfo,- writeMetadata: this.client.writeMetadata,- persistStorage: this.client.persistStorage,- },- this.client.logger,- );-- return datasetInfo;- }-}diff --git a/packages/memory-storage/src/resource-clients/dataset.ts b/packages/memory-storage/src/resource-clients/dataset.tsindex 716cea3d8..bbb3bdbf8 100644--- a/packages/memory-storage/src/resource-clients/dataset.ts+++ b/packages/memory-storage/src/resource-clients/dataset.ts@@ -56,7 +56,7 @@ export class DatasetClient<Data extends Dictionary = Dictionary>this.client = options.client;}- async get(): Promise<storage.DatasetInfo | undefined> {+ async getMetadata(): Promise<storage.DatasetInfo> {const found = await findOrCacheDatasetByPossibleId(this.client, this.name ?? this.id);if (found) {@@ -64,7 +64,7 @@ export class DatasetClient<Data extends Dictionary = Dictionary>return found.toDatasetInfo();}- return undefined;+ return this.toDatasetInfo();}async update(newFields: storage.DatasetClientUpdateOptions = {}): Promise<storage.DatasetInfo> {@@ -87,7 +87,7 @@ export class DatasetClient<Data extends Dictionary = Dictionary>}// Check that name is not in use already- const existingStoreByName = this.client.datasetClientsHandled.find(+ const existingStoreByName = this.client.datasetClientCache.find((store) => store.name?.toLowerCase() === parsed.name!.toLowerCase(),);@@ -113,10 +113,10 @@ export class DatasetClient<Data extends Dictionary = Dictionary>}async delete(): Promise<void> {- const storeIndex = this.client.datasetClientsHandled.findIndex((store) => store.id === this.id);+ const storeIndex = this.client.datasetClientCache.findIndex((store) => store.id === this.id);if (storeIndex !== -1) {- const [oldClient] = this.client.datasetClientsHandled.splice(storeIndex, 1);+ const [oldClient] = this.client.datasetClientCache.splice(storeIndex, 1);oldClient.itemCount = 0;oldClient.datasetEntries.clear();diff --git a/packages/memory-storage/src/resource-clients/key-value-store-collection.ts b/packages/memory-storage/src/resource-clients/key-value-store-collection.tsdeleted file mode 100644index 4db3b1e05..000000000--- a/packages/memory-storage/src/resource-clients/key-value-store-collection.ts+++ /dev/null@@ -1,74 +0,0 @@-import { resolve } from 'node:path';--import type * as storage from '@crawlee/types';-import { s } from '@sapphire/shapeshift';--import { scheduleBackgroundTask } from '../background-handler/index.js';-import { findOrCacheKeyValueStoreByPossibleId } from '../cache-helpers.js';-import type { MemoryStorage } from '../index.js';-import { KeyValueStoreClient } from './key-value-store.js';--export interface KeyValueStoreCollectionClientOptions {- baseStorageDirectory: string;- client: MemoryStorage;-}--export class KeyValueStoreCollectionClient implements storage.KeyValueStoreCollectionClient {- private readonly keyValueStoresDirectory: string;- private readonly client: MemoryStorage;-- constructor({ baseStorageDirectory, client }: KeyValueStoreCollectionClientOptions) {- this.keyValueStoresDirectory = resolve(baseStorageDirectory);- this.client = client;- }-- async list(): ReturnType<storage.KeyValueStoreCollectionClient['list']> {- return {- total: this.client.keyValueStoresHandled.length,- count: this.client.keyValueStoresHandled.length,- offset: 0,- limit: this.client.keyValueStoresHandled.length,- desc: false,- items: this.client.keyValueStoresHandled- .map((store) => store.toKeyValueStoreInfo())- .sort((a, b) => a.createdAt.getTime() - b.createdAt.getTime()),- };- }-- async getOrCreate(name?: string): Promise<storage.KeyValueStoreInfo> {- s.string().optional().parse(name);-- if (name) {- const found = await findOrCacheKeyValueStoreByPossibleId(this.client, name);-- if (found) {- return found.toKeyValueStoreInfo();- }- }-- const newStore = new KeyValueStoreClient({- name,- baseStorageDirectory: this.keyValueStoresDirectory,- client: this.client,- });- this.client.keyValueStoresHandled.push(newStore);-- // Schedule the worker to write to the disk- const kvStoreInfo = newStore.toKeyValueStoreInfo();-- scheduleBackgroundTask(- {- action: 'update-metadata',- entityType: 'keyValueStores',- entityDirectory: newStore.keyValueStoreDirectory,- id: kvStoreInfo.name ?? kvStoreInfo.id,- data: kvStoreInfo,- writeMetadata: this.client.writeMetadata,- persistStorage: this.client.persistStorage,- },- this.client.logger,- );-- return kvStoreInfo;- }-}diff --git a/packages/memory-storage/src/resource-clients/key-value-store.ts b/packages/memory-storage/src/resource-clients/key-value-store.tsindex 28cf34c17..f9f5fb0fd 100644--- a/packages/memory-storage/src/resource-clients/key-value-store.ts+++ b/packages/memory-storage/src/resource-clients/key-value-store.ts@@ -60,7 +60,7 @@ export class KeyValueStoreClient extends BaseClient {this.client = options.client;}- async get(): Promise<storage.KeyValueStoreInfo | undefined> {+ async getMetadata(): Promise<storage.KeyValueStoreInfo> {const found = await findOrCacheKeyValueStoreByPossibleId(this.client, this.name ?? this.id);if (found) {@@ -68,7 +68,7 @@ export class KeyValueStoreClient extends BaseClient {return found.toKeyValueStoreInfo();}- return undefined;+ return this.toKeyValueStoreInfo();}async update(newFields: storage.KeyValueStoreClientUpdateOptions = {}): Promise<storage.KeyValueStoreInfo> {@@ -91,7 +91,7 @@ export class KeyValueStoreClient extends BaseClient {}// Check that name is not in use already- const existingStoreByName = this.client.keyValueStoresHandled.find(+ const existingStoreByName = this.client.keyValueStoreCache.find((store) => store.name?.toLowerCase() === parsed.name!.toLowerCase(),);@@ -117,10 +117,10 @@ export class KeyValueStoreClient extends BaseClient {}async delete(): Promise<void> {- const storeIndex = this.client.keyValueStoresHandled.findIndex((store) => store.id === this.id);+ const storeIndex = this.client.keyValueStoreCache.findIndex((store) => store.id === this.id);if (storeIndex !== -1) {- const [oldClient] = this.client.keyValueStoresHandled.splice(storeIndex, 1);+ const [oldClient] = this.client.keyValueStoreCache.splice(storeIndex, 1);oldClient.keyValueEntries.clear();await rm(oldClient.keyValueStoreDirectory, { recursive: true, force: true });diff --git a/packages/memory-storage/src/resource-clients/request-queue-collection.ts b/packages/memory-storage/src/resource-clients/request-queue-collection.tsdeleted file mode 100644index 90ed3c81d..000000000--- a/packages/memory-storage/src/resource-clients/request-queue-collection.ts+++ /dev/null@@ -1,74 +0,0 @@-import { resolve } from 'node:path';--import type * as storage from '@crawlee/types';-import { s } from '@sapphire/shapeshift';--import { scheduleBackgroundTask } from '../background-handler/index.js';-import { findRequestQueueByPossibleId } from '../cache-helpers.js';-import type { MemoryStorage } from '../index.js';-import { RequestQueueClient } from './request-queue.js';--export interface RequestQueueCollectionClientOptions {- baseStorageDirectory: string;- client: MemoryStorage;-}--export class RequestQueueCollectionClient implements storage.RequestQueueCollectionClient {- private readonly requestQueuesDirectory: string;- private readonly client: MemoryStorage;-- constructor({ baseStorageDirectory, client }: RequestQueueCollectionClientOptions) {- this.requestQueuesDirectory = resolve(baseStorageDirectory);- this.client = client;- }-- async list(): ReturnType<storage.RequestQueueCollectionClient['list']> {- return {- total: this.client.requestQueuesHandled.length,- count: this.client.requestQueuesHandled.length,- offset: 0,- limit: this.client.requestQueuesHandled.length,- desc: false,- items: this.client.requestQueuesHandled- .map((store) => store.toRequestQueueInfo())- .sort((a, b) => a.createdAt.getTime() - b.createdAt.getTime()),- };- }-- async getOrCreate(name?: string): Promise<storage.RequestQueueInfo> {- s.string().optional().parse(name);-- if (name) {- const found = await findRequestQueueByPossibleId(this.client, name);-- if (found) {- return found.toRequestQueueInfo();- }- }-- const newStore = new RequestQueueClient({- name,- baseStorageDirectory: this.requestQueuesDirectory,- client: this.client,- });- this.client.requestQueuesHandled.push(newStore);-- // Schedule the worker to write to the disk- const queueInfo = newStore.toRequestQueueInfo();-- scheduleBackgroundTask(- {- action: 'update-metadata',- entityType: 'requestQueues',- entityDirectory: newStore.requestQueueDirectory,- id: queueInfo.name ?? queueInfo.id,- data: queueInfo,- writeMetadata: this.client.writeMetadata,- persistStorage: this.client.persistStorage,- },- this.client.logger,- );-- return queueInfo;- }-}diff --git a/packages/memory-storage/src/resource-clients/request-queue.ts b/packages/memory-storage/src/resource-clients/request-queue.tsindex aaa1f6c78..e4772c492 100644--- a/packages/memory-storage/src/resource-clients/request-queue.ts+++ b/packages/memory-storage/src/resource-clients/request-queue.ts@@ -85,7 +85,7 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQuereturn existingQueueById;}- async get(): Promise<storage.RequestQueueInfo | undefined> {+ async getMetadata(): Promise<storage.RequestQueueInfo> {const found = await findRequestQueueByPossibleId(this.client, this.name ?? this.id);if (found) {@@ -93,7 +93,7 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQuereturn found.toRequestQueueInfo();}- return undefined;+ return this.toRequestQueueInfo();}async update(newFields: { name?: string | undefined }): Promise<storage.RequestQueueInfo | undefined> {@@ -118,7 +118,7 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQue}// Check that name is not in use already- const existingQueueByName = this.client.requestQueuesHandled.find(+ const existingQueueByName = this.client.requestQueueCache.find((queue) => queue.name?.toLowerCase() === parsed.name!.toLowerCase(),);@@ -144,10 +144,10 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQue}async delete(): Promise<void> {- const storeIndex = this.client.requestQueuesHandled.findIndex((queue) => queue.id === this.id);+ const storeIndex = this.client.requestQueueCache.findIndex((queue) => queue.id === this.id);if (storeIndex !== -1) {- const [oldClient] = this.client.requestQueuesHandled.splice(storeIndex, 1);+ const [oldClient] = this.client.requestQueueCache.splice(storeIndex, 1);oldClient.pendingRequestCount = 0;oldClient.requests.clear();diff --git a/packages/memory-storage/test/async-iteration.test.ts b/packages/memory-storage/test/async-iteration.test.tsindex b3f512561..b3d9d7779 100644--- a/packages/memory-storage/test/async-iteration.test.ts+++ b/packages/memory-storage/test/async-iteration.test.ts@@ -23,8 +23,7 @@ describe('Async iteration support', () => {let dataset: DatasetClient;beforeAll(async () => {- const { id } = await storage.datasets().getOrCreate('async-iteration-dataset');- dataset = storage.dataset(id);+ dataset = await storage.createDatasetClient({ name: 'async-iteration-dataset' });await dataset.pushItems(elements);});@@ -98,8 +97,7 @@ describe('Async iteration support', () => {let kvStore: KeyValueStoreClient;beforeAll(async () => {- const { id } = await storage.keyValueStores().getOrCreate('async-iteration-kvs');- kvStore = storage.keyValueStore(id);+ kvStore = await storage.createKeyValueStoreClient({ name: 'async-iteration-kvs' });for (const key of keys) {await kvStore.setRecord({ key, value: { data: key } });@@ -127,8 +125,7 @@ describe('Async iteration support', () => {test('respects limit option when iterating (10 items, limit 2)', async () => {// Create a fresh store with exactly 10 items to match the reported bug scenario- const { id } = await storage.keyValueStores().getOrCreate('limit-test-kvs');- const testStore = storage.keyValueStore(id);+ const testStore = await storage.createKeyValueStoreClient({ name: 'limit-test-kvs' });for (let i = 0; i < 10; i++) {await testStore.setRecord({ key: `key-${i}`, value: `value-${i}` });@@ -175,8 +172,7 @@ describe('Async iteration support', () => {let kvStore: KeyValueStoreClient;beforeAll(async () => {- const { id } = await storage.keyValueStores().getOrCreate('async-iteration-kvs-keys');- kvStore = storage.keyValueStore(id);+ kvStore = await storage.createKeyValueStoreClient({ name: 'async-iteration-kvs-keys' });for (const key of keys) {await kvStore.setRecord({ key, value: { data: key } });@@ -262,8 +258,7 @@ describe('Async iteration support', () => {let kvStore: KeyValueStoreClient;beforeAll(async () => {- const { id } = await storage.keyValueStores().getOrCreate('async-iteration-kvs-values');- kvStore = storage.keyValueStore(id);+ kvStore = await storage.createKeyValueStoreClient({ name: 'async-iteration-kvs-values' });for (const key of keys) {await kvStore.setRecord({ key, value: { data: key } });@@ -348,8 +343,7 @@ describe('Async iteration support', () => {let kvStore: KeyValueStoreClient;beforeAll(async () => {- const { id } = await storage.keyValueStores().getOrCreate('async-iteration-kvs-entries');- kvStore = storage.keyValueStore(id);+ kvStore = await storage.createKeyValueStoreClient({ name: 'async-iteration-kvs-entries' });for (const key of keys) {await kvStore.setRecord({ key, value: { data: key } });@@ -489,8 +483,7 @@ describe('Async iteration support', () => {let kvStore: KeyValueStoreClient;beforeAll(async () => {- const { id } = await storage.keyValueStores().getOrCreate('lazy-test-kvs-values');- kvStore = storage.keyValueStore(id);+ kvStore = await storage.createKeyValueStoreClient({ name: 'lazy-test-kvs-values' });for (let i = 0; i < 5; i++) {await kvStore.setRecord({ key: `key-${i}`, value: { data: i } });@@ -530,8 +523,7 @@ describe('Async iteration support', () => {let kvStore: KeyValueStoreClient;beforeAll(async () => {- const { id } = await storage.keyValueStores().getOrCreate('lazy-test-kvs-entries');- kvStore = storage.keyValueStore(id);+ kvStore = await storage.createKeyValueStoreClient({ name: 'lazy-test-kvs-entries' });for (let i = 0; i < 5; i++) {await kvStore.setRecord({ key: `key-${i}`, value: { data: i } });diff --git a/packages/memory-storage/test/fs-fallback.test.ts b/packages/memory-storage/test/fs-fallback.test.tsindex 50edb321e..ca8ac5ff7 100644--- a/packages/memory-storage/test/fs-fallback.test.ts+++ b/packages/memory-storage/test/fs-fallback.test.ts@@ -55,8 +55,8 @@ describe('fallback to fs for reading', () => {// POST INIT //test('attempting to read "default" key value store with "__metadata__" present should read from fs', async () => {- const defaultStoreInfo = await storage.keyValueStores().getOrCreate('default');- const defaultStore = storage.keyValueStore(defaultStoreInfo.id);+ const defaultStore = await storage.createKeyValueStoreClient({ name: 'default' });+ const defaultStoreInfo = await defaultStore.getMetadata();expect(defaultStoreInfo.name).toEqual('default');expect(defaultStoreInfo.createdAt).toEqual(expectedFsDate);@@ -70,7 +70,7 @@ describe('fallback to fs for reading', () => {});test('attempting to read "other" key value store with no "__metadata__" present should read from fs, even if accessed without generating id first', async () => {- const otherStore = storage.keyValueStore('other');+ const otherStore = await storage.createKeyValueStoreClient({ name: 'other' });const input = await otherStore.getRecord('INPUT');expect(input).toStrictEqual<KeyValueStoreRecord>({@@ -80,12 +80,14 @@ describe('fallback to fs for reading', () => {});});- test('attempting to read non-existent "default_2" key value store should return undefined', async () => {- await expect(storage.keyValueStore('default_2').get()).resolves.toBeUndefined();+ test('attempting to read "default_2" key value store that has no data on disk should still be accessible after creation', async () => {+ const default2Store = await storage.createKeyValueStoreClient({ name: 'default_2' });+ const info = await default2Store.getMetadata();+ expect(info.name).toEqual('default_2');});test('attempting to read "no-ext" key value store should load the missing extension file correctly', async () => {- const noExtStore = storage.keyValueStore('no-ext');+ const noExtStore = await storage.createKeyValueStoreClient({ name: 'no-ext' });const input = await noExtStore.getRecord('INPUT');expect(input).toStrictEqual<KeyValueStoreRecord>({@@ -96,7 +98,7 @@ describe('fallback to fs for reading', () => {});test('attempting to read "invalid-json" key value store should ignore the invalid "INPUT" json file', async () => {- const invalidJsonStore = storage.keyValueStore('invalid-json');+ const invalidJsonStore = await storage.createKeyValueStoreClient({ name: 'invalid-json' });const input = await invalidJsonStore.getRecord('INPUT');expect(input).toBeUndefined();diff --git a/packages/memory-storage/test/key-value-store/stream.test.ts b/packages/memory-storage/test/key-value-store/stream.test.tsindex d0d7c7bff..8577a4266 100644--- a/packages/memory-storage/test/key-value-store/stream.test.ts+++ b/packages/memory-storage/test/key-value-store/stream.test.ts@@ -10,8 +10,7 @@ describe('KeyValueStore should drain streams when setting records', () => {const fsStream = Readable.from([Buffer.from('hello'), Buffer.from('world')]);test('should drain stream', async () => {- const defaultStoreInfo = await storage.keyValueStores().getOrCreate('default');- const defaultStore = storage.keyValueStore(defaultStoreInfo.id);+ const defaultStore = await storage.createKeyValueStoreClient({ name: 'default' });await defaultStore.setRecord({ key: 'streamz', value: fsStream, contentType: 'text/plain' });diff --git a/packages/memory-storage/test/no-crash-on-big-buffers.test.ts b/packages/memory-storage/test/no-crash-on-big-buffers.test.tsindex 5677d20ac..473bc0b3f 100644--- a/packages/memory-storage/test/no-crash-on-big-buffers.test.ts+++ b/packages/memory-storage/test/no-crash-on-big-buffers.test.ts@@ -5,7 +5,7 @@ import { rm } from 'node:fs/promises';import { resolve } from 'node:path';import { MemoryStorage } from '@crawlee/memory-storage';-import type { KeyValueStoreClient, KeyValueStoreInfo } from '@crawlee/types';+import type { KeyValueStoreClient } from '@crawlee/types';describe('MemoryStorage should not crash when saving a big buffer', () => {const tmpLocation = resolve(import.meta.dirname, './tmp/no-buffer-crash');@@ -14,12 +14,10 @@ describe('MemoryStorage should not crash when saving a big buffer', () => {persistStorage: false,});- let kvs: KeyValueStoreInfo;let store: KeyValueStoreClient;beforeAll(async () => {- kvs = await storage.keyValueStores().getOrCreate();- store = storage.keyValueStore(kvs.id);+ store = await storage.createKeyValueStoreClient();});afterAll(async () => {diff --git a/packages/memory-storage/test/no-writing-to-disk.test.ts b/packages/memory-storage/test/no-writing-to-disk.test.tsindex 83692d66d..b7cc86f52 100644--- a/packages/memory-storage/test/no-writing-to-disk.test.ts+++ b/packages/memory-storage/test/no-writing-to-disk.test.ts@@ -20,9 +20,7 @@ describe('persistStorage option', () => {});test('creating a key-value pair in a key-value store should not write data to the disk', async () => {- const keyValueStoreInfo = await storage.keyValueStores().getOrCreate();-- const keyValueStore = storage.keyValueStore(keyValueStoreInfo.id);+ const keyValueStore = await storage.createKeyValueStoreClient();await keyValueStore.setRecord({ key: 'foo', value: 'test' });// We check that reading the directory for the store throws an error, which means it wasn't created on disk@@ -39,11 +37,10 @@ describe('persistStorage option', () => {});test('creating a key-value pair in a key-value store should not write data to the disk, but it should write the __metadata__ file', async () => {- const keyValueStoreInfo = await storage.keyValueStores().getOrCreate();-- const keyValueStore = storage.keyValueStore(keyValueStoreInfo.id);+ const keyValueStore = await storage.createKeyValueStoreClient();await keyValueStore.setRecord({ key: 'foo', value: 'test' });+ const keyValueStoreInfo = await keyValueStore.getMetadata();const storePath = resolve(storage.keyValueStoresDirectory, keyValueStoreInfo.id);await waitTillWrittenToDisk(storePath);diff --git a/packages/memory-storage/test/request-queue/forefront.test.ts b/packages/memory-storage/test/request-queue/forefront.test.tsindex 33f477de9..45553b599 100644--- a/packages/memory-storage/test/request-queue/forefront.test.ts+++ b/packages/memory-storage/test/request-queue/forefront.test.ts@@ -11,8 +11,7 @@ describe('RequestQueueV1 respects `forefront` in `listHead`', () => {let requestQueue: RequestQueueClient;beforeEach(async () => {- const { id } = await storage.requestQueues().getOrCreate('forefront');- requestQueue = storage.requestQueue(id);+ requestQueue = await storage.createRequestQueueClient({ name: 'forefront' });});afterEach(async () => {@@ -199,8 +198,7 @@ describe('RequestQueueV2 respects `forefront` in `listAndLockHead`', () => {let requestQueue: RequestQueueClient;beforeEach(async () => {- const { id } = await storage.requestQueues().getOrCreate('forefront-v2');- requestQueue = storage.requestQueue(id);+ requestQueue = await storage.createRequestQueueClient({ name: 'forefront-v2' });});afterEach(async () => {diff --git a/packages/memory-storage/test/request-queue/handledRequestCount-should-update.test.ts b/packages/memory-storage/test/request-queue/handledRequestCount-should-update.test.tsindex 9bc0e5955..c711408ab 100644--- a/packages/memory-storage/test/request-queue/handledRequestCount-should-update.test.ts+++ b/packages/memory-storage/test/request-queue/handledRequestCount-should-update.test.ts@@ -9,8 +9,7 @@ describe('RequestQueue handledRequestCount should update', () => {let requestQueue: RequestQueueClient;beforeAll(async () => {- const { id } = await storage.requestQueues().getOrCreate('handledRequestCount');- requestQueue = storage.requestQueue(id);+ requestQueue = await storage.createRequestQueueClient({ name: 'handledRequestCount' });});test('after updating the request, it should increment the handledRequestCount', async () => {@@ -23,8 +22,8 @@ describe('RequestQueue handledRequestCount should update', () => {handledAt: new Date().toISOString(),});- const updatedStatistics = await requestQueue.get();- expect(updatedStatistics?.handledRequestCount).toEqual(1);+ const updatedStatistics = await requestQueue.getMetadata();+ expect(updatedStatistics.handledRequestCount).toEqual(1);});test('adding an already handled request should increment the handledRequestCount', async () => {@@ -34,8 +33,8 @@ describe('RequestQueue handledRequestCount should update', () => {handledAt: new Date().toISOString(),});- const updatedStatistics = await requestQueue.get();- expect(updatedStatistics?.handledRequestCount).toEqual(2);+ const updatedStatistics = await requestQueue.getMetadata();+ expect(updatedStatistics.handledRequestCount).toEqual(2);});test('deleting a request should decrement the handledRequestCount', async () => {@@ -47,8 +46,8 @@ describe('RequestQueue handledRequestCount should update', () => {await requestQueue.deleteRequest(requestId);- const updatedStatistics = await requestQueue.get();- expect(updatedStatistics?.handledRequestCount).toEqual(2);+ const updatedStatistics = await requestQueue.getMetadata();+ expect(updatedStatistics.handledRequestCount).toEqual(2);});test('updating an already handled request should not increment the handledRequestCount again', async () => {diff --git a/packages/memory-storage/test/request-queue/ignore-non-json-files.test.ts b/packages/memory-storage/test/request-queue/ignore-non-json-files.test.tsindex ee8543ee4..24bf710fd 100644--- a/packages/memory-storage/test/request-queue/ignore-non-json-files.test.ts+++ b/packages/memory-storage/test/request-queue/ignore-non-json-files.test.ts@@ -53,8 +53,8 @@ describe('when falling back to fs, Request queue should ignore non-JSON files',});test('attempting to list "default" request queue should ignore non-JSON files', async () => {- const defaultQueueInfo = await storage.requestQueues().getOrCreate('default');- const defaultQueue = storage.requestQueue(defaultQueueInfo.id);+ const defaultQueue = await storage.createRequestQueueClient({ name: 'default' });+ const defaultQueueInfo = await defaultQueue.getMetadata();expect(defaultQueueInfo.name).toEqual('default');diff --git a/packages/memory-storage/test/reverse-datataset-list.test.ts b/packages/memory-storage/test/reverse-datataset-list.test.tsindex 07cd85e9c..d28553795 100644--- a/packages/memory-storage/test/reverse-datataset-list.test.ts+++ b/packages/memory-storage/test/reverse-datataset-list.test.ts@@ -20,8 +20,7 @@ describe('Dataset#listItems respects the desc option', () => {});beforeAll(async () => {- const { id: falseDatasetId } = await storage.datasets().getOrCreate('false');- dataset = storage.dataset(falseDatasetId);+ dataset = await storage.createDatasetClient({ name: 'false' });await dataset.pushItems(elements);});diff --git a/packages/memory-storage/test/write-metadata.test.ts b/packages/memory-storage/test/write-metadata.test.tsindex a11e1726e..68a465a90 100644--- a/packages/memory-storage/test/write-metadata.test.ts+++ b/packages/memory-storage/test/write-metadata.test.ts@@ -20,19 +20,19 @@ describe('writeMetadata option', () => {});test('creating a data store should not write __metadata__.json file', async () => {- const keyValueStore = await storage.keyValueStores().getOrCreate();- const expectedPath = resolve(storage.keyValueStoresDirectory, keyValueStore.id);+ const keyValueStore = await storage.createKeyValueStoreClient();+ const info = await keyValueStore.getMetadata();+ const expectedPath = resolve(storage.keyValueStoresDirectory, info.id);// We check that reading the directory for the store throws an error, which means it wasn't created on diskawait expect(async () => readdir(expectedPath)).rejects.toThrow();});test('creating a key-value pair in a key-value store should not write __metadata__.json file for the value', async () => {- const keyValueStoreInfo = await storage.keyValueStores().getOrCreate();-- const keyValueStore = storage.keyValueStore(keyValueStoreInfo.id);+ const keyValueStore = await storage.createKeyValueStoreClient();await keyValueStore.setRecord({ key: 'foo', value: 'test' });+ const keyValueStoreInfo = await keyValueStore.getMetadata();const expectedFilePath = resolve(storage.keyValueStoresDirectory, `${keyValueStoreInfo.id}/foo.txt`);await waitTillWrittenToDisk(expectedFilePath);@@ -50,8 +50,9 @@ describe('writeMetadata option', () => {});test('creating a data store should write __metadata__.json file', async () => {- const keyValueStore = await storage.keyValueStores().getOrCreate();- const expectedPath = resolve(storage.keyValueStoresDirectory, keyValueStore.id);+ const keyValueStore = await storage.createKeyValueStoreClient();+ const info = await keyValueStore.getMetadata();+ const expectedPath = resolve(storage.keyValueStoresDirectory, info.id);await waitTillWrittenToDisk(expectedPath);const directoryFiles = await readdir(expectedPath);@@ -60,11 +61,10 @@ describe('writeMetadata option', () => {});test('creating a key-value pair in a key-value store should write __metadata__.json file for the value', async () => {- const keyValueStoreInfo = await storage.keyValueStores().getOrCreate();-- const keyValueStore = storage.keyValueStore(keyValueStoreInfo.id);+ const keyValueStore = await storage.createKeyValueStoreClient();await keyValueStore.setRecord({ key: 'foo', value: 'test' });+ const keyValueStoreInfo = await keyValueStore.getMetadata();const expectedFilePath = resolve(storage.keyValueStoresDirectory, `${keyValueStoreInfo.id}/foo.txt`);const expectedMetadataPath = resolve(storage.keyValueStoresDirectory,diff --git a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.tsindex 5bc81b02c..42320882e 100644--- a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts+++ b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts@@ -745,7 +745,7 @@ export class AdaptivePlaywrightCrawler<...calls.pushData.map(async (params) => crawlingContext.pushData(...params)),...calls.addRequests.map(async (params) => crawlingContext.addRequests(...params)),...Object.entries(keyValueStoreChanges).map(async ([storeIdOrName, changes]) => {- const store = await crawlingContext.getKeyValueStore(storeIdOrName);+ const store = await crawlingContext.getKeyValueStore({ id: storeIdOrName });await Promise.all(Object.entries(changes).map(async ([key, { changedValue, options }]) =>store.setValue(key, changedValue, options),diff --git a/packages/playwright-crawler/src/internals/utils/playwright-utils.ts b/packages/playwright-crawler/src/internals/utils/playwright-utils.tsindex 8665a198a..e86c69349 100644--- a/packages/playwright-crawler/src/internals/utils/playwright-utils.ts+++ b/packages/playwright-crawler/src/internals/utils/playwright-utils.ts@@ -563,7 +563,7 @@ export async function saveSnapshot(page: Page, options: SaveSnapshotOptions = {}} = options;try {- const store = await KeyValueStore.open(keyValueStoreName, {+ const store = await KeyValueStore.open(keyValueStoreName ? { name: keyValueStoreName } : null, {config: config ?? Configuration.getGlobalConfig(),});diff --git a/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.ts b/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.tsindex 27caf9f4f..d188e2d57 100644--- a/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.ts+++ b/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.ts@@ -775,7 +775,7 @@ export async function saveSnapshot(page: Page, options: SaveSnapshotOptions = {}} = options;try {- const store = await KeyValueStore.open(keyValueStoreName, {+ const store = await KeyValueStore.open(keyValueStoreName ? { name: keyValueStoreName } : null, {config: config ?? Configuration.getGlobalConfig(),});diff --git a/packages/types/src/storages.ts b/packages/types/src/storages.tsindex fc5879020..687e74e40 100644--- a/packages/types/src/storages.ts+++ b/packages/types/src/storages.ts@@ -15,18 +15,6 @@ export interface QueueOperationInfo {requestId: string;}-export interface DatasetCollectionClientOptions {- storageDir: string;-}--export interface DatasetCollectionData {- id: string;- name?: string;- createdAt: Date;- modifiedAt: Date;- accessedAt: Date;-}-export interface PaginatedList<Data> {/** Total count of entries in the dataset. */total: number;@@ -42,18 +30,6 @@ export interface PaginatedList<Data> {items: Data[];}-export interface Dataset extends DatasetCollectionData {- itemCount: number;-}--/**- * Dataset collection client.- */-export interface DatasetCollectionClient {- list(): Promise<PaginatedList<Dataset>>;- getOrCreate(name?: string): Promise<DatasetCollectionData>;-}-export interface DatasetClientUpdateOptions {name?: string;}@@ -82,7 +58,14 @@ export interface DatasetStats {}export interface DatasetClient<Data extends Dictionary = Dictionary> {- get(): Promise<DatasetInfo | undefined>;+ /**+ * Returns metadata about the dataset (id, name, timestamps, item count, etc.).+ *+ * Implementations should throw if the underlying storage no longer exists+ * (e.g. it was deleted externally). This method should never return stale data+ * for a storage that has been removed.+ */+ getMetadata(): Promise<DatasetInfo>;update(newFields: DatasetClientUpdateOptions): Promise<Partial<DatasetInfo>>;delete(): Promise<void>;downloadItems(...args: unknown[]): Promise<Buffer>;@@ -113,14 +96,6 @@ export interface KeyValueStoreInfo {stats?: KeyValueStoreStats;}-/**- * Key-value store collection client.- */-export interface KeyValueStoreCollectionClient {- list(): Promise<PaginatedList<KeyValueStoreInfo>>;- getOrCreate(name?: string): Promise<KeyValueStoreInfo>;-}-export interface KeyValueStoreRecord {key: string;value: any;@@ -166,7 +141,14 @@ export interface KeyValueStoreClientGetRecordOptions {* Key-value Store client.*/export interface KeyValueStoreClient {- get(): Promise<KeyValueStoreInfo | undefined>;+ /**+ * Returns metadata about the key-value store (id, name, timestamps, etc.).+ *+ * Implementations should throw if the underlying storage no longer exists+ * (e.g. it was deleted externally). This method should never return stale data+ * for a storage that has been removed.+ */+ getMetadata(): Promise<KeyValueStoreInfo>;update(newFields: KeyValueStoreClientUpdateOptions): Promise<Partial<KeyValueStoreInfo>>;delete(): Promise<void>;listKeys(@@ -207,14 +189,6 @@ export interface RequestQueueInfo {stats?: RequestQueueStats;}-/**- * Request queue collection client.- */-export interface RequestQueueCollectionClient {- list(): Promise<PaginatedList<RequestQueueInfo>>;- getOrCreate(name: string): Promise<RequestQueueInfo>;-}-export interface RequestQueueHeadItem {id: string;retryCount: number;@@ -302,7 +276,14 @@ export interface BatchAddRequestsResult {}export interface RequestQueueClient {- get(): Promise<RequestQueueInfo | undefined>;+ /**+ * Returns metadata about the request queue (id, name, timestamps, request counts, etc.).+ *+ * Implementations should throw if the underlying storage no longer exists+ * (e.g. it was deleted externally). This method should never return stale data+ * for a storage that has been removed.+ */+ getMetadata(): Promise<RequestQueueInfo>;update(newFields: { name?: string }): Promise<Partial<RequestQueueInfo> | undefined>;delete(): Promise<void>;listHead(options?: ListOptions): Promise<QueueHead>;@@ -316,26 +297,89 @@ export interface RequestQueueClient {deleteRequestLock(id: string, options?: DeleteRequestLockOptions): Promise<void>;}-export interface RequestQueueOptions {- clientKey?: string;- timeoutSecs?: number;-}-export interface SetStatusMessageOptions {isStatusMessageTerminal?: boolean;level?: 'DEBUG' | 'INFO' | 'WARNING' | 'ERROR';}/**- * Represents a storage capable of working with datasets, KV stores and request queues.+ * Identifies a storage by either its ID or its name, but not both.+ * If neither is provided, the default storage for the given type is used.+ */+export type StorageIdentifier =+ | { id: string; name?: never }+ | { id?: never; name: string }+ | { id?: never; name?: never };++/**+ * Options for creating a dataset client via {@apilink StorageClient.createDatasetClient}.+ */+export type CreateDatasetClientOptions = StorageIdentifier;++/**+ * Options for creating a key-value store client via {@apilink StorageClient.createKeyValueStoreClient}.+ */+export type CreateKeyValueStoreClientOptions = StorageIdentifier;++/**+ * Options for creating a request queue client via {@apilink StorageClient.createRequestQueueClient}.+ */+export type CreateRequestQueueClientOptions = StorageIdentifier & {+ /**+ * Client key for request locking.+ * TODO: This is an Apify-platform concern and should eventually be pushed down+ * into the Apify SDK's client implementation (aligning with crawlee-python).+ */+ clientKey?: string;+ /**+ * Timeout in seconds for request queue operations.+ * TODO: This is an Apify-platform concern and should eventually be pushed down+ * into the Apify SDK's client implementation (aligning with crawlee-python).+ */+ timeoutSecs?: number;+};++/**+ * Represents a storage backend capable of working with datasets, key-value stores and request queues.+ *+ * A new storage backend needs to implement 4 classes:+ * - `StorageClient` - the factory that creates sub-clients+ * - `DatasetClient` - operations on a single dataset+ * - `KeyValueStoreClient` - operations on a single key-value store+ * - `RequestQueueClient` - operations on a single request queue+ *+ * The `StorageClient` acts as an async factory: each `create*` method either opens an existing+ * storage or creates a new one, returning a sub-client bound to that storage instance.*/export interface StorageClient {- datasets(): DatasetCollectionClient;- dataset(id: string): DatasetClient;- keyValueStores(): KeyValueStoreCollectionClient;- keyValueStore(id: string): KeyValueStoreClient;- requestQueues(): RequestQueueCollectionClient;- requestQueue(id: string, options?: RequestQueueOptions): RequestQueueClient;+ /**+ * Create (or open) a dataset client.+ * If `id` is provided, opens the dataset with that ID.+ * If `name` is provided, opens an existing dataset with that name or creates a new one.+ * If neither is provided, opens or creates the default dataset.+ */+ createDatasetClient(options?: CreateDatasetClientOptions): Promise<DatasetClient>;+ /**+ * Create (or open) a key-value store client.+ * If `id` is provided, opens the key-value store with that ID.+ * If `name` is provided, opens an existing store with that name or creates a new one.+ * If neither is provided, opens or creates the default key-value store.+ */+ createKeyValueStoreClient(options?: CreateKeyValueStoreClientOptions): Promise<KeyValueStoreClient>;+ /**+ * Create (or open) a request queue client.+ * If `id` is provided, opens the request queue with that ID.+ * If `name` is provided, opens an existing queue with that name or creates a new one.+ * If neither is provided, opens or creates the default request queue.+ */+ createRequestQueueClient(options?: CreateRequestQueueClientOptions): Promise<RequestQueueClient>;+ /**+ * Check whether a storage with the given ID exists.+ *+ * Used internally to resolve ambiguous `idOrName` strings passed to `Dataset.open()`,+ * `KeyValueStore.open()`, and `RequestQueue.open()`.+ */+ storageExists?(id: string, type: 'Dataset' | 'KeyValueStore' | 'RequestQueue'): Promise<boolean>;purge?(): Promise<void>;teardown?(): Promise<void>;setStatusMessage?(message: string, options?: SetStatusMessageOptions): Promise<void>;diff --git a/test/core/crawlers/adaptive_playwright_crawler.test.ts b/test/core/crawlers/adaptive_playwright_crawler.test.tsindex 1bae3cba3..b17c2ca0b 100644--- a/test/core/crawlers/adaptive_playwright_crawler.test.ts+++ b/test/core/crawlers/adaptive_playwright_crawler.test.ts@@ -524,7 +524,7 @@ describe('AdaptivePlaywrightCrawler', () => {);await crawler.run();- const store = localStorageEmulator.getKeyValueStore();+ const store = await localStorageEmulator.getKeyValueStore();expect((await store.getRecord('1'))!.value).toEqual({ content: 42 });expect((await store.getRecord('2'))!.value).toEqual({ content: 42 });@@ -561,7 +561,7 @@ describe('AdaptivePlaywrightCrawler', () => {'Directly accessing storage in a request handler is not allowed in AdaptivePlaywrightCrawler',);- const store = localStorageEmulator.getKeyValueStore();+ const store = await localStorageEmulator.getKeyValueStore();expect(await store.getRecord('1')).toBeUndefined();});diff --git a/test/core/crawlers/basic_crawler.test.ts b/test/core/crawlers/basic_crawler.test.tsindex 4510a003d..0d3e63258 100644--- a/test/core/crawlers/basic_crawler.test.ts+++ b/test/core/crawlers/basic_crawler.test.ts@@ -969,10 +969,7 @@ describe('BasicCrawler', () => {];const processed: Dictionary<Request> = {};const requestList = await RequestList.open(null, sources);- const requestQueue = new RequestQueue(- { id: 'xxx', client: serviceLocator.getStorageClient() },- serviceLocator.getConfiguration(),- );+ const requestQueue = await RequestQueue.open({ id: 'xxx' });const requestHandler: RequestHandler = async ({ request }) => {await sleep(10);@@ -1043,10 +1040,7 @@ describe('BasicCrawler', () => {});test('should say that task is not ready requestList is not set and requestQueue is empty', async () => {- const requestQueue = new RequestQueue(- { id: 'xxx', client: serviceLocator.getStorageClient() },- serviceLocator.getConfiguration(),- );+ const requestQueue = await RequestQueue.open({ id: 'xxx' });requestQueue.isEmpty = async () => Promise.resolve(true);const crawler = new BasicCrawler({@@ -1059,10 +1053,7 @@ describe('BasicCrawler', () => {});test('should be possible to override isFinishedFunction and isTaskReadyFunction of underlying AutoscaledPool', async () => {- const requestQueue = new RequestQueue(- { id: 'xxx', client: serviceLocator.getStorageClient() },- serviceLocator.getConfiguration(),- );+ const requestQueue = await RequestQueue.open({ id: 'xxx' });const processed: Request[] = [];const queue: Request[] = [];let isFinished = false;@@ -1136,10 +1127,7 @@ describe('BasicCrawler', () => {});test('keepAlive', async () => {- const requestQueue = new RequestQueue(- { id: 'xxx', client: serviceLocator.getStorageClient() },- serviceLocator.getConfiguration(),- );+ const requestQueue = await RequestQueue.open({ id: 'xxx' });const processed: Request[] = [];const queue: Request[] = [];@@ -1240,10 +1228,7 @@ describe('BasicCrawler', () => {});test('should load handledRequestCount from storages', async () => {- const requestQueue = new RequestQueue(- { id: 'id', client: serviceLocator.getStorageClient() },- serviceLocator.getConfiguration(),- );+ const requestQueue = await RequestQueue.open({ id: 'id' });requestQueue.isEmpty = async () => false;requestQueue.isFinished = async () => false;diff --git a/test/core/crawlers/rendering_type_predictor.test.ts b/test/core/crawlers/rendering_type_predictor.test.tsindex c96c1c360..ac44fc089 100644--- a/test/core/crawlers/rendering_type_predictor.test.ts+++ b/test/core/crawlers/rendering_type_predictor.test.ts@@ -34,7 +34,7 @@ describe('RenderingTypePredictor', () => {predictor.storeResult(clientRequest, 'clientOnly');// Persist the state- const store = localStorageEmulator.getKeyValueStore();+ const store = await localStorageEmulator.getKeyValueStore();// eslint-disable-next-line dot-notationawait predictor['state'].persistState(); // Access private state for persistencediff --git a/test/core/enqueue_links/click_elements.test.ts b/test/core/enqueue_links/click_elements.test.tsindex cb4193d0d..ada448150 100644--- a/test/core/enqueue_links/click_elements.test.ts+++ b/test/core/enqueue_links/click_elements.test.ts@@ -2,7 +2,6 @@ import type { Server } from 'node:http';import type { RequestQueueOperationOptions, Source } from 'crawlee';import {- Configuration,launchPlaywright,launchPuppeteer,playwrightClickElements,@@ -10,7 +9,6 @@ import {puppeteerClickElements,puppeteerUtils,RequestQueue,- serviceLocator,} from 'crawlee';import type { Browser as PWBrowser, Page as PWPage } from 'playwright';// @ts-ignore This only throws when compiled against puppeteer 25+ (ESM only), we only import types, so its alllll gooooood@@ -25,11 +23,9 @@ function isPlaywrightBrowser(browser: PPBrowser | PWBrowser): browser is PWBrowsreturn (browser as PWBrowser).browserType !== undefined;}-const apifyClient = serviceLocator.getStorageClient();--function createRequestQueueMock() {+async function createRequestQueueMock() {const enqueued: Source[] = [];- const requestQueue = new RequestQueue({ id: 'xxx', client: apifyClient }, serviceLocator.getConfiguration());+ const requestQueue = await RequestQueue.open({ id: 'xxx' });// @ts-expect-error Override method for testingrequestQueue.addRequests = async function (requests) {@@ -88,7 +84,7 @@ testCases.forEach(({ caseName, launchBrowser, clickElements, utils }) => {});test('should work', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();const html = `<html><body>@@ -117,10 +113,7 @@ testCases.forEach(({ caseName, launchBrowser, clickElements, utils }) => {test('accepts forefront option', async () => {const addedRequests: { request: Source; options?: RequestQueueOperationOptions }[] = [];- const requestQueue = new RequestQueue(- { id: 'xxx', client: serviceLocator.getStorageClient() },- serviceLocator.getConfiguration(),- );+ const requestQueue = await RequestQueue.open({ id: 'xxx' });requestQueue.addRequests = async (requests, options) => {for await (const request of requests) {addedRequests.push({ request: typeof request === 'string' ? { url: request } : request, options });diff --git a/test/core/enqueue_links/enqueue_links.test.ts b/test/core/enqueue_links/enqueue_links.test.tsindex af1dd3793..74e8b6baf 100644--- a/test/core/enqueue_links/enqueue_links.test.ts+++ b/test/core/enqueue_links/enqueue_links.test.ts@@ -4,12 +4,10 @@ import { launchPlaywright } from '@crawlee/playwright';import type { RequestQueueOperationOptions, Source } from '@crawlee/puppeteer';import {browserCrawlerEnqueueLinks,- Configuration,EnqueueStrategy,launchPuppeteer,Request,RequestQueue,- serviceLocator,} from '@crawlee/puppeteer';import { type CheerioRoot, RobotsTxtFile } from '@crawlee/utils';import { load } from 'cheerio';@@ -19,8 +17,6 @@ import type { Browser as PuppeteerBrowser, Page as PuppeteerPage } from 'puppeteimport log from '@apify/log';-const apifyClient = serviceLocator.getStorageClient();-const HTML = `<html><head>@@ -48,9 +44,9 @@ const HTML = `</html>`;-function createRequestQueueMock() {+async function createRequestQueueMock() {const enqueued: Source[] = [];- const requestQueue = new RequestQueue({ id: 'xxx', client: apifyClient }, serviceLocator.getConfiguration());+ const requestQueue = await RequestQueue.open({ id: 'xxx' });// @ts-expect-error Override method for testingrequestQueue.addRequests = async function (requests) {@@ -93,7 +89,7 @@ describe('enqueueLinks()', () => {});test('works with item limit', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();await browserCrawlerEnqueueLinks({options: { limit: 3, selector: '.click', strategy: EnqueueStrategy.All },page,@@ -119,7 +115,7 @@ describe('enqueueLinks()', () => {});test('works with globs', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();const globs = ['https://example.com/**/*', { glob: '?(http|https)://cool.com/', method: 'POST' as const }];await browserCrawlerEnqueueLinks({@@ -155,7 +151,7 @@ describe('enqueueLinks()', () => {});test('does not throw with empty globs', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();const globs = ['https://example.com/**/*','',@@ -178,7 +174,7 @@ describe('enqueueLinks()', () => {});test('works with regexps', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();const regexps = [/^https:\/\/example\.com\/(\w|\/)+/,{ regexp: /^(http|https):\/\/cool\.com\//, method: 'POST' as const, userData: { label: 'COOL' } },@@ -216,7 +212,7 @@ describe('enqueueLinks()', () => {});test('works with skipNavigation', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();await browserCrawlerEnqueueLinks({options: {@@ -236,7 +232,7 @@ describe('enqueueLinks()', () => {});test('works with exclude glob', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();const globs = ['https://example.com/**/*', { glob: '?(http|https)://cool.com/', method: 'POST' as const }];const exclude = ['**/first'];@@ -274,7 +270,7 @@ describe('enqueueLinks()', () => {});test('works with exclude regexp', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();const globs = ['https://example.com/**/*', { glob: '?(http|https)://cool.com/', method: 'POST' as const }];const exclude = [/first/];@@ -312,7 +308,7 @@ describe('enqueueLinks()', () => {});test('works with pseudoUrls', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();const pseudoUrls = ['https://example.com/[(\\w|-|/)*]',{ purl: '[http|https]://cool.com/', method: 'POST' as const, userData: { label: 'COOL' } },@@ -350,7 +346,7 @@ describe('enqueueLinks()', () => {});test('throws with RegExp pseudoUrls', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();const pseudoUrls = [/https:\/\/example\.com\/(\w|-|\/)*/, /(http|https):\/\/cool\.com\//];@@ -366,7 +362,7 @@ describe('enqueueLinks()', () => {});test('works with undefined pseudoUrls[]', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();await browserCrawlerEnqueueLinks({options: { selector: '.click', strategy: EnqueueStrategy.All },@@ -395,7 +391,7 @@ describe('enqueueLinks()', () => {});test('throws with null pseudoUrls[]', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();await expect(browserCrawlerEnqueueLinks({// @ts-expect-error invalid input@@ -408,7 +404,7 @@ describe('enqueueLinks()', () => {});test('works with empty pseudoUrls[]', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();await browserCrawlerEnqueueLinks({options: { selector: '.click', pseudoUrls: [], strategy: EnqueueStrategy.All },page,@@ -436,7 +432,7 @@ describe('enqueueLinks()', () => {});test('throws with sparse pseudoUrls[]', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();const pseudoUrls = ['https://example.com/[(\\w|-|/)*]', null, '[http|https]://cool.com/'];await expect(@@ -452,7 +448,7 @@ describe('enqueueLinks()', () => {});test('correctly resolves relative URLs with default strategy of same-hostname', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();await browserCrawlerEnqueueLinks({options: { baseUrl: 'http://www.absolute.com/removethis/' },page,@@ -472,7 +468,7 @@ describe('enqueueLinks()', () => {});test('correctly resolves relative URLs with the strategy of same-domain', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();await browserCrawlerEnqueueLinks({options: { baseUrl: 'http://www.absolute.com/removethis/', strategy: EnqueueStrategy.SameDomain },page,@@ -496,7 +492,7 @@ describe('enqueueLinks()', () => {});test('correctly resolves relative URLs with the strategy of all', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();await browserCrawlerEnqueueLinks({options: { baseUrl: 'http://www.absolute.com/removethis/', strategy: EnqueueStrategy.All },page,@@ -540,7 +536,7 @@ describe('enqueueLinks()', () => {});test('correctly works with transformRequestFunction', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();const pseudoUrls = ['https://example.com/[(\\w|-|/)*]', '[http|https]://cool.com/'];@@ -590,7 +586,7 @@ describe('enqueueLinks()', () => {});test('works with globs', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();const globs = ['https://example.com/**/*',{ glob: '?(http|https)://cool.com/', method: 'POST' as const, userData: { label: 'COOL' } },@@ -628,7 +624,7 @@ describe('enqueueLinks()', () => {});test('does not throw with empty globs', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();const globs = ['https://example.com/**/*',{ glob: '?(http|https)://cool.com/', method: 'POST' as const, userData: { label: 'COOL' } },@@ -649,7 +645,7 @@ describe('enqueueLinks()', () => {});test('works with RegExps', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();const regexps = [/^https:\/\/example\.com\/(\w|\/)+/,{ regexp: /^(http|https):\/\/cool\.com\//, method: 'POST' as const, userData: { label: 'COOL' } },@@ -687,7 +683,7 @@ describe('enqueueLinks()', () => {});test('works with string pseudoUrls', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();const pseudoUrls = ['https://example.com/[(\\w|-|/)*]',{ purl: '[http|https]://cool.com/', method: 'POST' as const, userData: { label: 'COOL' } },@@ -726,7 +722,7 @@ describe('enqueueLinks()', () => {});test('throws with RegExp pseudoUrls', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();const pseudoUrls = [/https:\/\/example\.com\/(\w|-|\/)*/, /(http|https):\/\/cool\.com\//];await expect(@@ -741,7 +737,7 @@ describe('enqueueLinks()', () => {});test('works with undefined pseudoUrls[]', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();await cheerioCrawlerEnqueueLinks({options: { selector: '.click', strategy: EnqueueStrategy.All },$,@@ -769,7 +765,7 @@ describe('enqueueLinks()', () => {});test('throws with null pseudoUrls[]', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();await expect(cheerioCrawlerEnqueueLinks({// @ts-expect-error invalid input@@ -782,7 +778,7 @@ describe('enqueueLinks()', () => {});test('works with empty pseudoUrls[]', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();await cheerioCrawlerEnqueueLinks({options: { selector: '.click', pseudoUrls: [], strategy: EnqueueStrategy.All },$,@@ -810,7 +806,7 @@ describe('enqueueLinks()', () => {});test('throws with sparse pseudoUrls[]', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();const pseudoUrls = ['https://example.com/[(\\w|-|/)*]', null, '[http|https]://cool.com/'];await expect(@@ -826,7 +822,7 @@ describe('enqueueLinks()', () => {});test('correctly resolves relative URLs with the strategy of all', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();await cheerioCrawlerEnqueueLinks({options: { baseUrl: 'http://www.absolute.com/removethis/', strategy: EnqueueStrategy.All },$,@@ -870,7 +866,7 @@ describe('enqueueLinks()', () => {});test('correctly resolves relative URLs with the default strategy of same-hostname', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();await cheerioCrawlerEnqueueLinks({options: { baseUrl: 'http://www.absolute.com/removethis/' },$,@@ -890,7 +886,7 @@ describe('enqueueLinks()', () => {});test('correctly resolves relative URLs with the strategy of same-domain', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();await cheerioCrawlerEnqueueLinks({options: { baseUrl: 'http://www.absolute.com/removethis/', strategy: EnqueueStrategy.SameDomain },$,@@ -965,7 +961,7 @@ describe('enqueueLinks()', () => {});test('correctly resolves relative URLs with `urls` option', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();await cheerioCrawlerEnqueueLinks({options: {baseUrl: 'http://www.absolute.com/removethis/',@@ -988,7 +984,7 @@ describe('enqueueLinks()', () => {});test('correctly works with transformRequestFunction', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();const pseudoUrls = ['https://example.com/[(\\w|-|/)*]', '[http|https]://cool.com/'];await cheerioCrawlerEnqueueLinks({@@ -1026,10 +1022,7 @@ describe('enqueueLinks()', () => {test('accepts forefront option', async () => {const enqueued: { request: Source; options?: RequestQueueOperationOptions }[] = [];- const requestQueue = new RequestQueue(- { id: 'xxx', client: apifyClient },- serviceLocator.getConfiguration(),- );+ const requestQueue = await RequestQueue.open({ id: 'xxx' });requestQueue.addRequests = async (requests, options) => {// copy the requests to the enqueued list, along with options that were passed to addRequests,@@ -1058,10 +1051,7 @@ describe('enqueueLinks()', () => {test('accepts waitForAllRequestsToBeAdded option', async () => {const enqueued: { request: string | Source; options?: AddRequestsBatchedOptions }[] = [];- const requestQueue = new RequestQueue(- { id: 'xxx', client: apifyClient },- serviceLocator.getConfiguration(),- );+ const requestQueue = await RequestQueue.open({ id: 'xxx' });requestQueue.addRequestsBatched = async (requests, options) => {// copy the requests to the enqueued list, along with options that were passed to addRequests,@@ -1090,7 +1080,7 @@ describe('enqueueLinks()', () => {describe('label precedence', () => {test('global label option is applied if no other label is provided', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();await cheerioCrawlerEnqueueLinks({options: {@@ -1110,7 +1100,7 @@ describe('enqueueLinks()', () => {});test('pattern label overrides global label', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();await cheerioCrawlerEnqueueLinks({options: {@@ -1136,7 +1126,7 @@ describe('enqueueLinks()', () => {});test('transformRequestFunction has highest priority and overrides pattern label', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();await cheerioCrawlerEnqueueLinks({options: {@@ -1165,7 +1155,7 @@ describe('enqueueLinks()', () => {});test('transformRequestFunction can override all label sources', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();await cheerioCrawlerEnqueueLinks({options: {@@ -1195,7 +1185,7 @@ describe('enqueueLinks()', () => {});test('transformRequestFunction can modify other request properties after patterns are applied', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();await cheerioCrawlerEnqueueLinks({options: {@@ -1225,7 +1215,7 @@ describe('enqueueLinks()', () => {test('transformRequestFunction can return a new plain object instead of modifying in place', async () => {const enqueued: Source[] = [];- const requestQueue = new RequestQueue({ id: 'xxx', client: apifyClient });+ const requestQueue = await RequestQueue.open({ id: 'xxx' });// Custom mock that checks for Request instances - we override addRequestsBatched// to verify that request options returned by transformRequestFunction are converted to Request instances@@ -1272,7 +1262,7 @@ describe('enqueueLinks()', () => {});test('transformRequestFunction supports "skip" and "unchanged" string returns', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();const onSkippedRequest = vi.fn();await cheerioCrawlerEnqueueLinks({@@ -1315,7 +1305,7 @@ describe('enqueueLinks()', () => {});test('transformRequestFunction returning falsy correctly triggers onSkippedRequest', async () => {- const { enqueued, requestQueue } = createRequestQueueMock();+ const { enqueued, requestQueue } = await createRequestQueueMock();const onSkippedRequest = vi.fn();await cheerioCrawlerEnqueueLinks({diff --git a/test/core/recoverable_state.test.ts b/test/core/recoverable_state.test.tsindex 2a7b76e75..947ea7bff 100644--- a/test/core/recoverable_state.test.ts+++ b/test/core/recoverable_state.test.ts@@ -179,7 +179,9 @@ describe('RecoverableState', () => {recoverableState.currentValue.data.value = 'updated';await recoverableState.persistState();- const persistedState = JSON.parse((await localStorageEmulator.getKeyValueStore().getRecord('test-key'))?.value);+ const persistedState = JSON.parse(+ (await (await localStorageEmulator.getKeyValueStore()).getRecord('test-key'))?.value,+ );expect(persistedState).toMatchObject({data: { value: 'updated' },});diff --git a/test/core/session_pool/session_pool.test.ts b/test/core/session_pool/session_pool.test.tsindex 3a092d78b..7eb522f56 100644--- a/test/core/session_pool/session_pool.test.ts+++ b/test/core/session_pool/session_pool.test.ts@@ -315,7 +315,7 @@ describe('SessionPool - testing session pool', () => {await newSessionPool.teardown();// @ts-expect-error private symbol- const kvStore = await KeyValueStore.open(newSessionPool.persistStateKeyValueStoreId);+ const kvStore = await KeyValueStore.open({ id: newSessionPool.persistStateKeyValueStoreId });// @ts-expect-error private symbolconst state = await kvStore.getValue(newSessionPool.persistStateKey);diff --git a/test/core/storages/dataset.test.ts b/test/core/storages/dataset.test.tsindex 77470d084..c4021cf9b 100644--- a/test/core/storages/dataset.test.ts+++ b/test/core/storages/dataset.test.ts@@ -15,7 +15,10 @@ afterAll(async () => {});describe('dataset', () => {- const storageClient = serviceLocator.getStorageClient();+ async function createDataset(id = 'some-id', name?: string) {+ const client = await serviceLocator.getStorageClient().createDatasetClient(name ? { name } : { id });+ return new Dataset({ id, name, client });+ }beforeEach(async () => {vitest.clearAllMocks();@@ -25,10 +28,7 @@ describe('dataset', () => {const mockData = (bytes: number) => 'x'.repeat(bytes);test('should work', async () => {- const dataset = new Dataset({- id: 'some-id',- client: storageClient,- });+ const dataset = await createDataset();const pushItemSpy = vitest.spyOn(dataset.client, 'pushItems');@@ -57,10 +57,7 @@ describe('dataset', () => {test('should successfully save large data', async () => {const half = mockData(MAX_PAYLOAD_SIZE_BYTES / 2);- const dataset = new Dataset({- id: 'some-id',- client: storageClient,- });+ const dataset = await createDataset();const mockPushItems = vitest.spyOn(dataset.client, 'pushItems');mockPushItems.mockResolvedValueOnce(undefined);@@ -81,10 +78,7 @@ describe('dataset', () => {const expectedFirst = JSON.stringify(Array(count - 1).fill(chunk));const expectedSecond = JSON.stringify([chunk]);- const dataset = new Dataset({- id: 'some-id',- client: storageClient,- });+ const dataset = await createDataset();const mockPushItems = vitest.spyOn(dataset.client, 'pushItems');mockPushItems.mockResolvedValueOnce(undefined);@@ -99,7 +93,7 @@ describe('dataset', () => {test('should throw on too large file', async () => {const full = mockData(MAX_PAYLOAD_SIZE_BYTES);- const dataset = new Dataset({ id: 'some-id', client: storageClient });+ const dataset = await createDataset();try {await dataset.pushData({ foo: full });throw new Error('Should fail!');@@ -110,7 +104,7 @@ describe('dataset', () => {});test('should throw on too large file in an array', async () => {const full = mockData(MAX_PAYLOAD_SIZE_BYTES);- const dataset = new Dataset({ id: 'some-id', client: storageClient });+ const dataset = await createDataset();try {await dataset.pushData([{ foo: 0 }, { foo: 1 }, { foo: 2 }, { foo: full }, { foo: 4 }]);throw new Error('Should fail!');@@ -121,10 +115,7 @@ describe('dataset', () => {});test('getData() should work', async () => {- const dataset = new Dataset({- id: 'some-id',- client: storageClient,- });+ const dataset = await createDataset();const expected = {items: [{ foo: 'bar' }, { foo: 'hotel' }],@@ -161,7 +152,7 @@ describe('dataset', () => {});test('getInfo() should work', async () => {- const dataset = new Dataset({ id: 'some-id', client: storageClient });+ const dataset = await createDataset();const expected: Awaited<ReturnType<Dataset['getInfo']>> = {id: 'WkzbQMuFYuamGv3YF',@@ -172,7 +163,7 @@ describe('dataset', () => {itemCount: 14,};- const mockGetDataset = vitest.spyOn(dataset.client, 'get');+ const mockGetDataset = vitest.spyOn(dataset.client, 'getMetadata');mockGetDataset.mockResolvedValueOnce(expected);const result = await dataset.getInfo();@@ -180,11 +171,8 @@ describe('dataset', () => {expect(result).toEqual(expected);});- const getRemoteDataset = () => {- const dataset = new Dataset({- id: 'some-id',- client: storageClient,- });+ const getRemoteDataset = async () => {+ const dataset = await createDataset();const firstResolve = {items: [{ foo: 'a' }, { foo: 'b' }],@@ -224,7 +212,7 @@ describe('dataset', () => {};test('forEach() should work', async () => {- const { dataset, restoreAndVerify } = getRemoteDataset();+ const { dataset, restoreAndVerify } = await getRemoteDataset();const items: Dictionary[] = [];const indexes: number[] = [];@@ -245,7 +233,7 @@ describe('dataset', () => {});test('map() should work', async () => {- const { dataset, restoreAndVerify } = getRemoteDataset();+ const { dataset, restoreAndVerify } = await getRemoteDataset();const result = await dataset.map((item, index) => {@@ -267,7 +255,7 @@ describe('dataset', () => {});test('map() should support promises', async () => {- const { dataset, restoreAndVerify } = getRemoteDataset();+ const { dataset, restoreAndVerify } = await getRemoteDataset();const result = await dataset.map(async (item, index) => {@@ -290,7 +278,7 @@ describe('dataset', () => {});test('reduce() should work', async () => {- const { dataset, restoreAndVerify } = getRemoteDataset();+ const { dataset, restoreAndVerify } = await getRemoteDataset();const result = await dataset.reduce((memo, item, index) => {@@ -316,7 +304,7 @@ describe('dataset', () => {});test('reduce() should support promises', async () => {- const { dataset, restoreAndVerify } = getRemoteDataset();+ const { dataset, restoreAndVerify } = await getRemoteDataset();const result = await dataset.reduce(async (memo, item, index) => {@@ -342,11 +330,7 @@ describe('dataset', () => {});test('reduce() uses first value as memo if no memo is provided', async () => {- const dataset = new Dataset({- id: 'some-id',- name: 'some-name',- client: storageClient,- });+ const dataset = await createDataset('some-id', 'some-name');const mockListItems = vitest.spyOn(dataset.client, 'listItems');mockListItems.mockResolvedValueOnce({items: [{ foo: 4 }, { foo: 5 }],@@ -395,10 +379,7 @@ describe('dataset', () => {describe('pushData', () => {test('throws on invalid args', async () => {- const dataset = new Dataset({- id: 'some-id',- client: storageClient,- });+ const dataset = await createDataset();// @ts-expect-error JS-side validationawait expect(dataset.pushData()).rejects.toThrow('Expected `data` to be of type `object` but received type `undefined`',@@ -501,7 +482,7 @@ describe('dataset', () => {];it('Should work', async () => {- const dataset = await Dataset.open(Math.random().toString(36));+ const dataset = await Dataset.open({ name: Math.random().toString(36) });await dataset.pushData(dataToPush);await dataset.exportToJSON('HELLO');@@ -535,7 +516,7 @@ describe('dataset', () => {];it('Should work', async () => {- const dataset = await Dataset.open(Math.random().toString(36));+ const dataset = await Dataset.open({ name: Math.random().toString(36) });await dataset.pushData(dataToPush);await dataset.exportToCSV('HELLO-csv');@@ -561,7 +542,7 @@ describe('dataset', () => {await dataset.exportTo('test.csv',{- toKVS: kvStore.name,+ toKVS: { name: kvStore.name! },collectAllKeys: true,},'text/csv',diff --git a/test/core/storages/key_value_store.test.ts b/test/core/storages/key_value_store.test.tsindex dc78c4eba..d28d39b54 100644--- a/test/core/storages/key_value_store.test.ts+++ b/test/core/storages/key_value_store.test.ts@@ -15,17 +15,17 @@ afterAll(async () => {});describe('KeyValueStore', () => {- const client = serviceLocator.getStorageClient();+ async function createKeyValueStore(id = 'some-id-1', name?: string) {+ const client = await serviceLocator.getStorageClient().createKeyValueStoreClient(name ? { name } : { id });+ return new KeyValueStore({ id, name, client });+ }beforeEach(async () => {vitest.clearAllMocks();});test('should work', async () => {- const store = new KeyValueStore({- id: 'some-id-1',- client,- });+ const store = await createKeyValueStore();// Record definitionconst record = { foo: 'bar' };@@ -105,10 +105,7 @@ describe('KeyValueStore', () => {describe('getValue', () => {test('throws on invalid args', async () => {- const store = new KeyValueStore({- id: 'some-id-1',- client,- });+ const store = await createKeyValueStore();// @ts-expect-error JS-side validationawait expect(store.getValue()).rejects.toThrow(@@ -143,10 +140,7 @@ describe('KeyValueStore', () => {describe('recordExists', () => {test('throws on invalid args', async () => {- const store = new KeyValueStore({- id: 'some-id-1',- client,- });+ const store = await createKeyValueStore();// @ts-expect-error JS-side validationawait expect(store.recordExists()).rejects.toThrow(@@ -176,10 +170,7 @@ describe('KeyValueStore', () => {describe('setValue', () => {test('throws on invalid args', async () => {- const store = new KeyValueStore({- id: 'some-id-1',- client,- });+ const store = await createKeyValueStore();// @ts-expect-error JS-side validationawait expect(store.setValue()).rejects.toThrow(@@ -255,10 +246,7 @@ describe('KeyValueStore', () => {});test('throws on invalid key', async () => {- const store = new KeyValueStore({- id: 'my-store-id',- client,- });+ const store = await createKeyValueStore('my-store-id');const INVALID_CHARACTERS = '?|\\/"*<>%:';for (const char of INVALID_CHARACTERS) {@@ -274,10 +262,7 @@ describe('KeyValueStore', () => {});test('correctly adds charset to content type', async () => {- const store = new KeyValueStore({- id: 'my-store-id-1',- client,- });+ const store = await createKeyValueStore('my-store-id-1');const mockSetRecord = vitest// @ts-expect-error Accessing private property@@ -301,10 +286,7 @@ describe('KeyValueStore', () => {});test('correctly passes object values as JSON', async () => {- const store = new KeyValueStore({- id: 'my-store-id-1',- client,- });+ const store = await createKeyValueStore('my-store-id-1');const record = { foo: 'bar' };const recordStr = JSON.stringify(record, null, 2);@@ -331,10 +313,7 @@ describe('KeyValueStore', () => {});test('correctly passes timeout options', async () => {- const store = new KeyValueStore({- id: 'my-store-id-1',- client,- });+ const store = await createKeyValueStore('my-store-id-1');const record = { foo: 'bar' };const recordStr = JSON.stringify(record, null, 2);@@ -364,10 +343,7 @@ describe('KeyValueStore', () => {});test('correctly passes raw string values', async () => {- const store = new KeyValueStore({- id: 'my-store-id-1',- client,- });+ const store = await createKeyValueStore('my-store-id-1');const mockSetRecord = vitest// @ts-expect-error Accessing private property@@ -391,10 +367,7 @@ describe('KeyValueStore', () => {});test('correctly passes raw Buffer values', async () => {- const store = new KeyValueStore({- id: 'my-store-id-1',- client,- });+ const store = await createKeyValueStore('my-store-id-1');const mockSetRecord = vitest// @ts-expect-error Accessing private property@@ -419,10 +392,7 @@ describe('KeyValueStore', () => {});test('correctly passes a stream', async () => {- const store = new KeyValueStore({- id: 'my-store-id-1',- client,- });+ const store = await createKeyValueStore('my-store-id-1');const mockSetRecord = vitest// @ts-expect-error Accessing private property@@ -543,10 +513,7 @@ describe('KeyValueStore', () => {});test('should work remotely', async () => {- const store = new KeyValueStore({- id: 'my-store-id-1',- client,- });+ const store = await createKeyValueStore('my-store-id-1');// @ts-expect-error Accessing private propertyconst mockListKeys = vitest.spyOn(store.client, 'listKeys');diff --git a/test/core/storages/request_queue.test.ts b/test/core/storages/request_queue.test.tsindex 5e7c27cda..31881f15e 100644--- a/test/core/storages/request_queue.test.ts+++ b/test/core/storages/request_queue.test.ts@@ -38,14 +38,17 @@ beforeEach(async () => {});describe('RequestQueue remote', () => {- const storageClient = serviceLocator.getStorageClient();+ async function createRequestQueue(id = 'some-id', name?: string) {+ const client = await serviceLocator.getStorageClient().createRequestQueueClient(name ? { name } : { id });+ return new RequestQueue({ id, name, client }, serviceLocator.getConfiguration());+ }beforeEach(() => {vitest.clearAllMocks();});test('should work', async () => {- const queue = new RequestQueue({ id: 'some-id', client: storageClient }, serviceLocator.getConfiguration());+ const queue = await createRequestQueue();const firstResolveValue = {requestId: 'a',wasAlreadyHandled: false,@@ -202,10 +205,7 @@ describe('RequestQueue remote', () => {});test('addRequests', async () => {- const queue = new RequestQueue(- { id: 'batch-requests', client: storageClient },- serviceLocator.getConfiguration(),- );+ const queue = await createRequestQueue('batch-requests');const mockAddRequests = vitest.spyOn(queue.client, 'batchAddRequests');const requestOptions = { url: 'http://example.com/a' };@@ -354,7 +354,7 @@ describe('RequestQueue remote', () => {});test('should cache new requests locally', async () => {- const queue = new RequestQueue({ id: 'some-id', client: storageClient }, serviceLocator.getConfiguration());+ const queue = await createRequestQueue();const requestA = new Request({ url: 'http://example.com/a' });const requestB = new Request({ url: 'http://example.com/a' }); // Has same uniqueKey as A@@ -385,7 +385,7 @@ describe('RequestQueue remote', () => {});test('should cache requests locally with info if request was already handled', async () => {- const queue = new RequestQueue({ id: 'some-id', client: storageClient }, serviceLocator.getConfiguration());+ const queue = await createRequestQueue();const requestX = new Request({ url: 'http://example.com/x' });const requestY = new Request({ url: 'http://example.com/x' }); // Has same uniqueKey as X@@ -416,7 +416,7 @@ describe('RequestQueue remote', () => {});test('should cache requests from queue head', async () => {- const queue = new RequestQueue({ id: 'some-id', client: storageClient }, serviceLocator.getConfiguration());+ const queue = await createRequestQueue();// Query queue head with request Aconst listHeadMock = vitest.spyOn(queue.client, 'listHead');@@ -444,10 +444,7 @@ describe('RequestQueue remote', () => {});test('should handle situation when newly created request is not available yet', async () => {- const queue = new RequestQueue(- { id: 'some-id', name: 'some-queue', client: storageClient },- serviceLocator.getConfiguration(),- );+ const queue = await createRequestQueue('some-id', 'some-queue');const listHeadMock = vitest.spyOn(queue.client, 'listHead');const requestA = new Request({ url: 'http://example.com/a' });@@ -498,7 +495,7 @@ describe('RequestQueue remote', () => {});test('should not add handled request to queue head dict', async () => {- const queue = new RequestQueue({ id: 'some-id', client: storageClient }, serviceLocator.getConfiguration());+ const queue = await createRequestQueue();const requestA = new Request({ url: 'http://example.com/a' });@@ -528,7 +525,7 @@ describe('RequestQueue remote', () => {});test('should accept plain object in addRequest()', async () => {- const queue = new RequestQueue({ id: 'some-id', client: storageClient }, serviceLocator.getConfiguration());+ const queue = await createRequestQueue();const addRequestMock = vitest.spyOn(queue.client, 'addRequest');addRequestMock.mockResolvedValueOnce({requestId: 'xxx',@@ -543,8 +540,8 @@ describe('RequestQueue remote', () => {});test('should return correct handledCount', async () => {- const queue = new RequestQueue({ id: 'id', client: storageClient }, serviceLocator.getConfiguration());- const getMock = vitest.spyOn(queue.client, 'get');+ const queue = await createRequestQueue('id');+ const getMock = vitest.spyOn(queue.client, 'getMetadata');getMock.mockResolvedValueOnce({handledRequestCount: 33,} as never);@@ -555,10 +552,7 @@ describe('RequestQueue remote', () => {});test('should always wait for a queue head to become consistent before marking queue as finished (hadMultipleClients = true)', async () => {- const queue = new RequestQueue(- { id: 'some-id', name: 'some-name', client: storageClient },- serviceLocator.getConfiguration(),- );+ const queue = await createRequestQueue('some-id', 'some-name');// Return head with modifiedAt = now so it will retry the call.const listHeadMock = vitest.spyOn(queue.client, 'listHead');@@ -584,10 +578,7 @@ describe('RequestQueue remote', () => {test('should always wait for a queue head to become consistent before marking queue as finished (hadMultipleClients = false)', async () => {const queueId = 'some-id';- const queue = new RequestQueue(- { id: queueId, name: 'some-name', client: storageClient },- serviceLocator.getConfiguration(),- );+ const queue = await createRequestQueue(queueId, 'some-name');expect(queue.assumedTotalCount).toBe(0);expect(queue.assumedHandledCount).toBe(0);@@ -753,10 +744,7 @@ describe('RequestQueue remote', () => {});test('getInfo() should work', async () => {- const queue = new RequestQueue(- { id: 'some-id', name: 'some-name', client: storageClient },- serviceLocator.getConfiguration(),- );+ const queue = await createRequestQueue('some-id', 'some-name');const expected = {id: 'WkzbQMuFYuamGv3YF',@@ -772,7 +760,7 @@ describe('RequestQueue remote', () => {hadMultipleClients: false,};- const getMock = vitest.spyOn(queue.client, 'get').mockResolvedValueOnce(expected);+ const getMock = vitest.spyOn(queue.client, 'getMetadata').mockResolvedValueOnce(expected);const result = await queue.getInfo();expect(result).toEqual(expected);@@ -781,10 +769,7 @@ describe('RequestQueue remote', () => {});test('drop() works', async () => {- const queue = new RequestQueue(- { id: 'some-id', name: 'some-name', client: storageClient },- serviceLocator.getConfiguration(),- );+ const queue = await createRequestQueue('some-id', 'some-name');const deleteMock = vitest.spyOn(queue.client, 'delete').mockResolvedValueOnce(undefined);await queue.drop();@@ -975,9 +960,9 @@ describe('RequestQueue v2', () => {}async function getEmptyQueue(name: string) {- const queue = await RequestQueueV2.open(name);+ const queue = await RequestQueueV2.open({ name });await queue.drop();- return RequestQueueV2.open(name);+ return RequestQueueV2.open({ name });}function getUniqueRequests(count: number) {diff --git a/test/shared/MemoryStorageEmulator.ts b/test/shared/MemoryStorageEmulator.tsindex c81876efe..9012070e2 100644--- a/test/shared/MemoryStorageEmulator.ts+++ b/test/shared/MemoryStorageEmulator.ts@@ -31,30 +31,33 @@ export class MemoryStorageEmulator extends StorageEmulator {}getDataset(id?: string) {- return this.storage.dataset(id ?? Configuration.getGlobalConfig().get('defaultDatasetId'));+ const resolvedId = id ?? Configuration.getGlobalConfig().get('defaultDatasetId');+ return this.storage.createDatasetClient(resolvedId ? { id: resolvedId } : {});}async getDatasetItems(id?: string) {- const dataset = this.getDataset(id);+ const dataset = await this.getDataset(id);return (await dataset.listItems()).items;}getRequestQueue(id?: string) {- return this.storage.requestQueue(id ?? Configuration.getGlobalConfig().get('defaultRequestQueueId'));+ const resolvedId = id ?? Configuration.getGlobalConfig().get('defaultRequestQueueId');+ return this.storage.createRequestQueueClient(resolvedId ? { id: resolvedId } : {});}async getRequestQueueItems(id?: string) {- const requestQueue = this.getRequestQueue(id);+ const requestQueue = await this.getRequestQueue(id);const { items: heads } = await requestQueue.listHead();return heads;}getKeyValueStore(id?: string) {- return this.storage.keyValueStore(id ?? Configuration.getGlobalConfig().get('defaultKeyValueStoreId'));+ const resolvedId = id ?? Configuration.getGlobalConfig().get('defaultKeyValueStoreId');+ return this.storage.createKeyValueStoreClient(resolvedId ? { id: resolvedId } : {});}async getState() {- return await this.getKeyValueStore().getRecord('CRAWLEE_STATE');+ return await (await this.getKeyValueStore()).getRecord('CRAWLEE_STATE');}}