Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
refactor!: Align `DatasetClient` and `KeyValueStoreClient` interfaces with their Python counterparts (#3627)
Jan Buchar59ec997ca910
diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.mdindex e2ca2c4a1..5080405e4 100644--- a/docs/upgrading/upgrading_v4.md+++ b/docs/upgrading/upgrading_v4.md@@ -465,10 +465,61 @@ The `StorageClient` interface changed from synchronous sub-client getters to **a| `client.requestQueue(id, opts)` | `client.createRequestQueueClient({ id?, name?, clientKey?, timeoutSecs? })` || `client.requestQueues().getOrCreate(name)` | _(absorbed into `createRequestQueueClient`)_ |-The `get()` method on `DatasetClient`, `KeyValueStoreClient`, and `RequestQueueClient` has been renamed to **`getMetadata()`**.+The sub-client interfaces (`DatasetClient`, `KeyValueStoreClient`, `RequestQueueClient`) have been aligned with their Python counterparts:++| Before (v3) | After (v4) |+|---|---|+| `get()` | `getMetadata()` |+| `update()` | Removed |+| `delete()` | `drop()` |+| _(n/a)_ | `purge()` (new — clears data, keeps storage) |++**`DatasetClient`:**++| Before (v3) | After (v4) |+|---|---|+| `pushItems(items: Data \| Data[] \| string \| string[])` | `pushData(items: Data[])` |+| `listItems(options?)` (dual iterable) | `getData(options?)` (returns a single `PaginatedList` page) |+| `listEntries(options?)` | Removed (handled by `Dataset` frontend) |+| `downloadItems()` | Removed |++**`KeyValueStoreClient`:**++| Before (v3) | After (v4) |+|---|---|+| `getRecord(key, options?)` | `getValue(key)` |+| `setRecord(record, options?)` | `setValue(record)` |+| `deleteRecord(key)` | `deleteValue(key)` |+| `getRecordPublicUrl(key)` | `getPublicUrl(key)` |+| `listKeys(options?)` → `KeyValueStoreClientListData` | `listKeys(options?)` → `KeyValueStoreItemData[]` |+| `keys()`, `values()`, `entries()` | Removed (handled by `KeyValueStore` frontend) |++**`RequestQueueClient`:**++| Before (v3) | After (v4) |+|---|---|+| `deleteRequest(id)` | Removed |++**Removed types** from `@crawlee/types`: `DatasetClientUpdateOptions`, `KeyValueStoreClientUpdateOptions`, `KeyValueStoreRecordOptions`, `KeyValueStoreClientListData`, `KeyValueStoreClientGetRecordOptions`. `KeyValueStoreClientListOptions` was renamed to `KeyValueStoreListKeysOptions`.The high-level storage classes (`Dataset`, `KeyValueStore`, `RequestQueue`) now receive their sub-client directly in the constructor options instead of receiving a `StorageClient` and calling its methods.+### `RecordOptions` simplified++`timeoutSecs` and `doNotRetryTimeouts` were removed from `RecordOptions` (used by `KeyValueStore.setValue`). Only `contentType` remains.++### `KeyValueStoreIteratorOptions` simplified++`exclusiveStartKey` and `collection` were removed. Only `prefix` remains.++### `Dataset.listItems` replaced by `Dataset.getData` and `Dataset.values`++`Dataset.listItems()` is replaced by two methods:+- `Dataset.getData(options?)` — returns a single `PaginatedList<Data>` page.+- `Dataset.values(options?)` — dual iterable: `for await...of` iterates all items; `await` returns all items as `Data[]`.++`Dataset.entries()` works the same way as `values()` but yields `[index, Data]` tuples. `KeyValueStore.keys()`, `.values()`, `.entries()` follow the same dual-iterable pattern.+### Removed `list()` methodThe `list()` method on collection clients (e.g. `client.datasets().list()`) has no replacement. If you were using it to enumerate all storages, you will need to use the Apify API client directly.@@ -479,7 +530,7 @@ If you implemented a custom `StorageClient`, you need to:1. Remove your `*CollectionClient` classes.2. Replace the six getter methods (`dataset`, `datasets`, `keyValueStore`, `keyValueStores`, `requestQueue`, `requestQueues`) with three async factory methods (`createDatasetClient`, `createKeyValueStoreClient`, `createRequestQueueClient`). Each factory should handle both opening an existing storage and creating a new one.-3. Rename `get()` to `getMetadata()` on your `DatasetClient`, `KeyValueStoreClient`, and `RequestQueueClient` implementations.+3. Apply the sub-client renames listed above (`get` → `getMetadata`, `delete` → `drop`, etc.) and implement the new `purge()` method.## Storage `.open()` now also accepts `{ id?, name? }`diff --git a/packages/core/src/crawlers/statistics.ts b/packages/core/src/crawlers/statistics.tsindex 29aaae3f5..8056c2344 100644--- a/packages/core/src/crawlers/statistics.ts+++ b/packages/core/src/crawlers/statistics.ts@@ -330,14 +330,8 @@ export class Statistics {this.log.debug('Persisting state', { persistStateKey: this.persistStateKey });- // use half the interval of `persistState` to avoid race conditions- const persistStateIntervalMillis = serviceLocator.getConfiguration().persistStateIntervalMillis;- const timeoutSecs = persistStateIntervalMillis / 2_000;await this.keyValueStore- .setValue(this.persistStateKey, this.toJSON(), {- timeoutSecs,- doNotRetryTimeouts: true,- })+ .setValue(this.persistStateKey, this.toJSON()).catch((error) =>this.log.warning(`Failed to persist the statistics to ${this.persistStateKey}`, { error }),);diff --git a/packages/core/src/session_pool/session_pool.ts b/packages/core/src/session_pool/session_pool.tsindex fb06659d5..f79d3f8b8 100644--- a/packages/core/src/session_pool/session_pool.ts+++ b/packages/core/src/session_pool/session_pool.ts@@ -371,14 +371,8 @@ export class SessionPool implements ISessionPool {persistStateKey: this.persistStateKey,});- // use half the interval of `persistState` to avoid race conditions- const persistStateIntervalMillis = serviceLocator.getConfiguration().persistStateIntervalMillis;- const timeoutSecs = persistStateIntervalMillis / 2_000;await this.keyValueStore- ?.setValue(this.persistStateKey, await this.getState(), {- timeoutSecs,- doNotRetryTimeouts: true,- })+ ?.setValue(this.persistStateKey, await this.getState()).catch((error) =>this.log.warning(`Failed to persist the session pool stats to ${this.persistStateKey}`, { error }),);diff --git a/packages/core/src/storages/dataset.ts b/packages/core/src/storages/dataset.tsindex 9b371450d..3f04d0bc7 100644--- a/packages/core/src/storages/dataset.ts+++ b/packages/core/src/storages/dataset.ts@@ -2,8 +2,6 @@ import type { DatasetClient, DatasetInfo, Dictionary, PaginatedList } from '@craimport { stringify } from 'csv-stringify/sync';import ow from 'ow';-import { MAX_PAYLOAD_SIZE_BYTES } from '@apify/consts';-import { Configuration } from '../configuration.js';import type { CrawleeLogger } from '../log.js';import { serviceLocator } from '../service_locator.js';@@ -13,20 +11,20 @@ import { KeyValueStore } from './key_value_store.js';import type { StorageIdentifier } from './storage_instance_manager.js';import type { StorageOpenOptions } from './utils.js';import { resolveStorageIdentifier } from './storage_instance_manager.js';-import { purgeDefaultStorages } from './utils.js';+import { createDualIterable, purgeDefaultStorages } from './utils.js';/** @internal */export const DATASET_ITERATORS_DEFAULT_LIMIT = 10000;-const SAFETY_BUFFER_PERCENT = 0.01 / 100; // 0.01%-/**- * Accepts a JSON serializable object as an input, validates its serializability,- * and validates its serialized size against limitBytes. Optionally accepts its index- * in an array to provide better error messages. Returns serialized object.+ * Validates that the given value is a plain JSON-serializable object+ * (not an array, not a primitive, not circular).+ *+ * @param item The value to validate.+ * @param index Optional index for error messages when validating inside an array.* @ignore*/-export function checkAndSerialize<T>(item: T, limitBytes: number, index?: number): string {+export function assertJsonSerializable<T>(item: T, index?: number): void {const s = typeof index === 'number' ? ` at index ${index} ` : ' ';const isItemObject = item && typeof item === 'object' && !Array.isArray(item);@@ -34,61 +32,12 @@ export function checkAndSerialize<T>(item: T, limitBytes: number, index?: numberthrow new Error(`Data item${s}is not an object. You can push only objects into a dataset.`);}- let payload;try {- payload = JSON.stringify(item);+ JSON.stringify(item);} catch (e) {const err = e as Error;throw new Error(`Data item${s}is not serializable to JSON.\nCause: ${err.message}`);}-- const bytes = Buffer.byteLength(payload);- if (bytes > limitBytes) {- throw new Error(`Data item${s}is too large (size: ${bytes} bytes, limit: ${limitBytes} bytes)`);- }-- return payload;-}--/**- * Takes an array of JSONs (payloads) as input and produces an array of JSON strings- * where each string is a JSON array of payloads with a maximum size of limitBytes per one- * JSON array. Fits as many payloads as possible into a single JSON array and then moves- * on to the next, preserving item order.- *- * The function assumes that none of the items is larger than limitBytes and does not validate.- * @ignore- */-export function chunkBySize(items: string[], limitBytes: number): string[] {- if (!items.length) return [];- if (items.length === 1) return items;-- // Split payloads into buckets of valid size.- let lastChunkBytes = 2; // Add 2 bytes for [] wrapper.- const chunks: (string | string[])[] = [];-- for (const payload of items) {- const bytes = Buffer.byteLength(payload);-- if (bytes <= limitBytes && bytes + 2 > limitBytes) {- // Handle cases where wrapping with [] would fail, but solo object is fine.- chunks.push(payload);- lastChunkBytes = bytes;- } else if (lastChunkBytes + bytes <= limitBytes) {- // ensure array- if (!Array.isArray(chunks[chunks.length - 1])) {- chunks.push([]);- }- (chunks[chunks.length - 1] as string[]).push(payload);- lastChunkBytes += bytes + 1; // Add 1 byte for ',' separator.- } else {- chunks.push([payload]);- lastChunkBytes = bytes + 2; // Add 2 bytes for [] wrapper.- }- }-- // Stringify array chunks.- return chunks.map((chunk) => (typeof chunk === 'string' ? chunk : `[${chunk.join(',')}]`));}export interface DatasetDataOptions {@@ -260,44 +209,21 @@ export class Dataset<Data extends Dictionary = Dictionary> {* **IMPORTANT**: Make sure to use the `await` keyword when calling `pushData()`,* otherwise the crawler process might finish before the data is stored!*- * The size of the data is limited by the receiving API and therefore `pushData()` will only- * allow objects whose JSON representation is smaller than 9MB. When an array is passed,- * none of the included objects- * may be larger than 9MB, but the array itself may be of any size.- *- * The function internally- * chunks the array into separate items and pushes them sequentially.- * The chunking process is stable (keeps order of data), but it does not provide a transaction- * safety mechanism. Therefore, in the event of an uploading error (after several automatic retries),- * the function's Promise will reject and the dataset will be left in a state where some of- * the items have already been saved to the dataset while other items from the source array were not.- * To overcome this limitation, the developer may, for example, read the last item saved in the dataset- * and re-attempt the save of the data from this item onwards to prevent duplicates.* @param data Object or array of objects containing data to be stored in the default dataset.- * The objects must be serializable to JSON and the JSON representation of each object must be smaller than 9MB.+ * The objects must be serializable to JSON.*/async pushData(data: Data | Data[]): Promise<void> {checkStorageAccess();ow(data, 'data', ow.object);- const dispatch = async (payload: string) => this.client.pushItems(payload);- const limit = MAX_PAYLOAD_SIZE_BYTES - Math.ceil(MAX_PAYLOAD_SIZE_BYTES * SAFETY_BUFFER_PERCENT);-- // Handle singular Objects- if (!Array.isArray(data)) {- const payload = checkAndSerialize(data, limit);- await dispatch(payload);- return;- }-- // Handle Arrays- const payloads = data.map((item, index) => checkAndSerialize(item, limit, index));- const chunks = chunkBySize(payloads, limit);- // Invoke client in series to preserve order of data- for (const chunk of chunks) {- await dispatch(chunk);+ // Normalize to array and validate each item+ const items = Array.isArray(data) ? data : [data];+ for (let i = 0; i < items.length; i++) {+ assertJsonSerializable(items[i], i);}++ await this.client.pushData(items);}/**@@ -307,7 +233,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {checkStorageAccess();try {- return await this.client.listItems(options);+ return await this.client.getData(options);} catch (e) {const error = e as Error;if (error.message.includes('Cannot create a string longer than')) {@@ -328,22 +254,9 @@ export class Dataset<Data extends Dictionary = Dictionary> {const items: Data[] = [];- const fetchNextChunk = async (offset = 0): Promise<void> => {- const limit = 1000;- const value = await this.client.listItems({ offset, limit, ...options });-- if (value.count === 0) {- return;- }-- items.push(...value.items);-- if (value.total > offset + value.count) {- await fetchNextChunk(offset + value.count);- }- };-- await fetchNextChunk();+ for await (const page of this.fetchPages(options)) {+ items.push(...page.items);+ }return items;}@@ -602,60 +515,99 @@ export class Dataset<Data extends Dictionary = Dictionary> {return currentMemo;}+ private async *fetchEntryPages(options: DatasetIteratorOptions): AsyncGenerator<PaginatedList<[number, Data]>> {+ let index = options.offset ?? 0;+ for await (const page of this.fetchPages(options)) {+ yield {+ ...page,+ items: page.items.map((item) => [index++, item] as [number, Data]),+ };+ }+ }++ private async *fetchPages(+ options: DatasetIteratorOptions,+ pageSize = DATASET_ITERATORS_DEFAULT_LIMIT,+ ): AsyncGenerator<PaginatedList<Data>> {+ let offset = options.offset ?? 0;+ const totalLimit = options.limit;+ let yielded = 0;++ while (true) {+ const fetchLimit = totalLimit !== undefined ? Math.min(pageSize, totalLimit - yielded) : pageSize;+ if (fetchLimit <= 0) break;++ const page = await this.client.getData({ ...options, offset, limit: fetchLimit });+ yield page;++ yielded += page.items.length;+ if (page.items.length < fetchLimit || offset + page.items.length >= page.total) break;+ offset += page.items.length;+ }+ }+/**- * Iterates over dataset items using an async generator,- * allowing the use of `for await...of` syntax.+ * Returns dataset items.+ *+ * When awaited (`await dataset.values()`), returns all items as a flat `Data[]` array.+ * When used as an async iterable (`for await...of`), iterates over all items across pages+ * without loading everything into memory at once.** **Example usage:*** ```javascript* const dataset = await Dataset.open('my-results');+ *+ * // Iterate over all items (memory-efficient for large datasets)* for await (const item of dataset.values()) {* console.log(item);* }+ *+ * // Or fetch all items at once+ * const items = await dataset.values();+ * console.log(items);* ```** @param options Options for the iteration.*/- values(options: DatasetIteratorOptions = {}): AsyncIterable<Data> & Promise<PaginatedList<Data>> {+ values(options: DatasetIteratorOptions = {}): AsyncIterable<Data> & Promise<Data[]> {checkStorageAccess();- const result = this.client.listItems(options) as AsyncIterable<Data> & Promise<PaginatedList<Data>>;-- if (!(Symbol.asyncIterator in result)) {- Object.defineProperty(result, Symbol.asyncIterator, {- get() {- throw new Error('Resource client "listItems" method does not return an async iterable.');- },- });- }-- return result;+ return createDualIterable({+ createPages: () => this.fetchPages(options),+ extractItems: (page) => page.items,+ });}/**- * Iterates over dataset entries (index-value pairs) using an async generator,- * allowing the use of `for await...of` syntax.+ * Returns dataset entries (index-value pairs).+ *+ * When awaited (`await dataset.entries()`), returns all entries as a flat `[index, item][]` array.+ * When used as an async iterable (`for await...of`), iterates over all entries across pages+ * without loading everything into memory at once.** **Example usage:*** ```javascript* const dataset = await Dataset.open('my-results');+ *+ * // Iterate over all entries* for await (const [index, item] of dataset.entries()) {* console.log(`Item at ${index}: ${JSON.stringify(item)}`);* }+ *+ * // Or fetch all at once+ * const entries = await dataset.entries();+ * console.log(entries);* ```** @param options Options for the iteration.*/- entries(- options: DatasetIteratorOptions = {},- ): AsyncIterable<[number, Data]> & Promise<PaginatedList<[number, Data]>> {+ entries(options: DatasetIteratorOptions = {}): AsyncIterable<[number, Data]> & Promise<[number, Data][]> {checkStorageAccess();- if (!this.client.listEntries) {- throw new Error('Resource client is missing the "listEntries" method.');- }-- return this.client.listEntries(options);+ return createDualIterable({+ createPages: () => this.fetchEntryPages(options),+ extractItems: (page) => page.items,+ });}/**@@ -681,7 +633,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {async drop(): Promise<void> {checkStorageAccess();- await this.client.delete();+ await this.client.drop();serviceLocator.getStorageInstanceManager().removeFromCache(this);}diff --git a/packages/core/src/storages/key_value_store.ts b/packages/core/src/storages/key_value_store.tsindex e04866a63..c81ced1da 100644--- a/packages/core/src/storages/key_value_store.ts+++ b/packages/core/src/storages/key_value_store.ts@@ -1,7 +1,7 @@import { readFile } from 'node:fs/promises';import { join } from 'node:path';-import type { Dictionary, KeyValueStoreClient } from '@crawlee/types';+import type { Dictionary, KeyValueStoreClient, KeyValueStoreItemData } from '@crawlee/types';import JSON5 from 'json5';import ow, { ArgumentError } from 'ow';@@ -15,7 +15,10 @@ import { checkStorageAccess } from './access_checking.js';import type { StorageIdentifier } from './storage_instance_manager.js';import type { StorageOpenOptions } from './utils.js';import { resolveStorageIdentifier } from './storage_instance_manager.js';-import { purgeDefaultStorages } from './utils.js';+import { createDualIterable, purgeDefaultStorages } from './utils.js';++/** @internal */+const KVS_KEYS_DEFAULT_LIMIT = 1000;/*** Helper function to possibly stringify value if options.contentType is not set.@@ -227,7 +230,7 @@ export class KeyValueStore {checkStorageAccess();ow(key, ow.string.nonEmpty);- const record = await this.client.getRecord(key);+ const record = await this.client.getValue(key);return (record?.value as T) ?? defaultValue ?? null;}@@ -273,19 +276,12 @@ export class KeyValueStore {return;}- // use half the interval of `persistState` to avoid race conditions- const persistStateIntervalMillis = this.config.persistStateIntervalMillis;- const timeoutSecs = persistStateIntervalMillis / 2_000;-serviceLocator.getEventManager().on('persistState', async () => {const promises: Promise<void>[] = [];for (const [key, value] of this.cache) {promises.push(- this.setValue(key, value, {- timeoutSecs,- doNotRetryTimeouts: true,- }).catch((error) =>+ this.setValue(key, value).catch((error) =>serviceLocator.getLogger().warning(`Failed to persist the state value to ${key}`, { error }),),);@@ -297,6 +293,34 @@ export class KeyValueStore {this.persistStateEventStarted = true;}+ private async *fetchKeyValuePages<T>(+ options: KeyValueStoreIteratorOptions,+ mapRecord: (key: string, value: unknown) => T,+ ): AsyncGenerator<T[]> {+ for await (const page of this.fetchKeyPages(options)) {+ const results: T[] = [];+ for (const item of page) {+ const record = await this.client.getValue(item.key);+ if (record) results.push(mapRecord(item.key, record.value));+ }+ yield results;+ }+ }++ private async *fetchKeyPages(+ options: KeyValueStoreIteratorOptions,+ limit = KVS_KEYS_DEFAULT_LIMIT,+ ): AsyncGenerator<KeyValueStoreItemData[]> {+ let exclusiveStartKey: string | undefined;++ while (true) {+ const items = await this.client.listKeys({ ...options, exclusiveStartKey, limit });+ yield items;+ if (items.length < limit) break;+ exclusiveStartKey = items[items.length - 1].key;+ }+ }+/*** Saves or deletes a record in the key-value store.* The function returns a promise that resolves once the record has been saved or deleted.@@ -367,8 +391,6 @@ export class KeyValueStore {options,ow.object.exactShape({contentType: ow.optional.string.nonEmpty,- timeoutSecs: ow.optional.number,- doNotRetryTimeouts: ow.optional.boolean,}),);@@ -393,21 +415,15 @@ export class KeyValueStore {}// In this case delete the record.- if (value === null) return this.client.deleteRecord(key);+ if (value === null) return this.client.deleteValue(key);value = maybeStringify(value, optionsCopy);- return this.client.setRecord(- {- key,- value,- contentType: optionsCopy.contentType,- },- {- timeoutSecs: optionsCopy.timeoutSecs,- doNotRetryTimeouts: optionsCopy.doNotRetryTimeouts,- },- );+ return this.client.setValue({+ key,+ value,+ contentType: optionsCopy.contentType,+ });}/**@@ -417,7 +433,7 @@ export class KeyValueStore {async drop(): Promise<void> {checkStorageAccess();- await this.client.delete();+ await this.client.drop();serviceLocator.getStorageInstanceManager().removeFromCache(this);}@@ -432,7 +448,7 @@ export class KeyValueStore {* Iterates over key-value store keys, yielding each in turn to an `iteratee` function.* Each invocation of `iteratee` is called with three arguments: `(key, index, info)`, where `key`* is the record key, `index` is a zero-based index of the key in the current iteration- * (regardless of `options.exclusiveStartKey`) and `info` is an object that contains a single property `size`+ * and `info` is an object that contains a single property `size`* indicating size of the record in bytes.** If the `iteratee` function returns a Promise then it is awaited before the next call.@@ -452,69 +468,74 @@ export class KeyValueStore {async forEachKey(iteratee: KeyConsumer, options: KeyValueStoreIteratorOptions = {}): Promise<void> {checkStorageAccess();- return this._forEachKey(iteratee, options);- }-- private async _forEachKey(- iteratee: KeyConsumer,- options: KeyValueStoreIteratorOptions = {},- index = 0,- ): Promise<void> {- const { exclusiveStartKey, prefix, collection } = options;ow(iteratee, ow.function);ow(options,ow.object.exactShape({- exclusiveStartKey: ow.optional.string,prefix: ow.optional.string,- collection: ow.optional.string,}),);- const response = await this.client.listKeys({ exclusiveStartKey, prefix, collection });- const { nextExclusiveStartKey, isTruncated, items } = response;- for (const item of items) {- await iteratee(item.key, index++, { size: item.size });+ let index = 0;++ for await (const page of this.fetchKeyPages(options)) {+ for (const item of page) {+ await iteratee(item.key, index++, { size: item.size });+ }}- return isTruncated- ? this._forEachKey(iteratee, { exclusiveStartKey: nextExclusiveStartKey, prefix, collection }, index)- : undefined; // [].forEach() returns undefined.}/**- * Iterates over key-value store keys using an async generator,- * allowing the use of `for await...of` syntax.+ * Returns key-value store keys.+ *+ * When awaited (`await store.keys()`), returns all keys as a flat `string[]` array.+ * When used as an async iterable (`for await...of`), iterates over all keys across pages+ * without loading everything into memory at once.** **Example usage:*** ```javascript* const keyValueStore = await KeyValueStore.open();+ *+ * // Iterate over all keys (memory-efficient for large stores)* for await (const key of keyValueStore.keys()) {* console.log(key);* }+ *+ * // Or fetch all keys at once+ * const allKeys = await keyValueStore.keys();+ * console.log(allKeys);* ```** @param options Options for the iteration.*/- async *keys(options: KeyValueStoreIteratorOptions = {}): AsyncGenerator<string, void, undefined> {+ keys(options: KeyValueStoreIteratorOptions = {}): AsyncIterable<string> & Promise<string[]> {checkStorageAccess();- if (!this.client.keys) {- throw new Error('Resource client is missing the "keys" method.');- }-- yield* this.client.keys(options);+ return createDualIterable({+ createPages: () => this.fetchKeyPages(options),+ extractItems: (page) => page.map((item) => item.key),+ });}/**- * Iterates over key-value store values using an async generator,- * allowing the use of `for await...of` syntax.+ * Returns key-value store values.+ *+ * When awaited (`await store.values()`), returns all values as a flat `T[]` array.+ * When used as an async iterable (`for await...of`), iterates over all values across pages+ * without loading everything into memory at once.** **Example usage:*** ```javascript* const keyValueStore = await KeyValueStore.open();+ *+ * // Iterate over all values (memory-efficient for large stores)* for await (const value of keyValueStore.values()) {* console.log(value);* }+ *+ * // Or fetch all values at once+ * const allValues = await keyValueStore.values();+ * console.log(allValues);* ```** @param options Options for the iteration.@@ -522,23 +543,31 @@ export class KeyValueStore {values<T = unknown>(options: KeyValueStoreIteratorOptions = {}): AsyncIterable<T> & Promise<T[]> {checkStorageAccess();- if (!this.client.values) {- throw new Error('Resource client is missing the "values" method.');- }-- return this.client.values(options) as AsyncIterable<T> & Promise<T[]>;+ return createDualIterable({+ createPages: () => this.fetchKeyValuePages<T>(options, (_key, value) => value as T),+ extractItems: (page) => page,+ });}/**- * Iterates over key-value store entries (key-value pairs) using an async generator,- * allowing the use of `for await...of` syntax.+ * Returns key-value store entries (key-value pairs).+ *+ * When awaited (`await store.entries()`), returns all entries as a flat `[key, value][]` array.+ * When used as an async iterable (`for await...of`), iterates over all entries across pages+ * without loading everything into memory at once.** **Example usage:*** ```javascript* const keyValueStore = await KeyValueStore.open();+ *+ * // Iterate over all entries (memory-efficient for large stores)* for await (const [key, value] of keyValueStore.entries()) {* console.log(`${key}: ${value}`);* }+ *+ * // Or fetch all entries at once+ * const allEntries = await keyValueStore.entries();+ * console.log(allEntries);* ```** @param options Options for the iteration.@@ -548,11 +577,10 @@ export class KeyValueStore {): AsyncIterable<[string, T]> & Promise<[string, T][]> {checkStorageAccess();- if (!this.client.entries) {- throw new Error('Resource client is missing the "entries" method.');- }-- return this.client.entries(options) as AsyncIterable<[string, T]> & Promise<[string, T][]>;+ return createDualIterable({+ createPages: () => this.fetchKeyValuePages<[string, T]>(options, (key, value) => [key, value as T]),+ extractItems: (page) => page,+ });}/**@@ -579,7 +607,7 @@ export class KeyValueStore {* @param key The key of the record to generate the public URL for.*/async getPublicUrl(key: string): Promise<string | undefined> {- return this.client.getRecordPublicUrl(key);+ return this.client.getPublicUrl(key);}/**@@ -842,29 +870,11 @@ export interface RecordOptions {* Specifies a custom MIME content type of the record.*/contentType?: string;-- /**- * Specifies a custom timeout for the `set-record` API call, in seconds.- */- timeoutSecs?: number;-- /**- * If set to `true`, the `set-record` API call will not be retried if it times out.- */- doNotRetryTimeouts?: boolean;}export interface KeyValueStoreIteratorOptions {- /**- * All keys up to this one (including) are skipped from the result.- */- exclusiveStartKey?: string;/*** If set, only keys that start with this prefix are returned.*/prefix?: string;- /**- * Collection name to use for listing keys.- */- collection?: string;}diff --git a/packages/core/src/storages/request_provider.ts b/packages/core/src/storages/request_provider.tsindex bf002fd1f..6b2ee8f9e 100644--- a/packages/core/src/storages/request_provider.ts+++ b/packages/core/src/storages/request_provider.ts@@ -804,7 +804,7 @@ export abstract class RequestProvider implements IStorage, IRequestManager {async drop(): Promise<void> {checkStorageAccess();- await this.client.delete();+ await this.client.drop();serviceLocator.getStorageInstanceManager().removeFromCache(this);}diff --git a/packages/core/src/storages/utils.ts b/packages/core/src/storages/utils.tsindex fbdb98132..bb7631076 100644--- a/packages/core/src/storages/utils.ts+++ b/packages/core/src/storages/utils.ts@@ -151,6 +151,77 @@ export const API_PROCESSED_REQUESTS_DELAY_MILLIS = 10_000;*/export const MAX_QUERIES_FOR_CONSISTENCY = 6;+/** @internal */+export interface DualIterableOptions<TItem, TRawPage> {+ /** Factory that returns an async generator yielding pages. */+ createPages: () => AsyncGenerator<TRawPage>;+ /** Extracts individual items from a page (for iteration). */+ extractItems: (page: TRawPage) => TItem[];+}++/**+ * Creates an object that is both an `AsyncIterable<TItem>` (for `for await...of`)+ * and a `Promise<TItem[]>` (for `await`) from a single async page generator.+ *+ * - `await result` drains all pages from a fresh generator and returns every+ * item as a flat array.+ * - `for await (const item of result)` streams all items across all pages,+ * yielding them one by one without buffering everything in memory.+ *+ * Each usage path creates its own generator instance, so `await` and+ * `for await...of` never interfere with each other.+ *+ * @internal+ */+export function createDualIterable<TItem, TRawPage>(+ options: DualIterableOptions<TItem, TRawPage>,+): AsyncIterable<TItem> & Promise<TItem[]> {+ const { createPages, extractItems } = options;+ let cached: Promise<TItem[]> | null = null;++ function getOrCreate(): Promise<TItem[]> {+ if (!cached) {+ cached = (async () => {+ const items: TItem[] = [];+ for await (const page of createPages()) {+ items.push(...extractItems(page));+ }+ return items;+ })();+ }+ return cached;+ }++ async function* iterateAll(): AsyncGenerator<TItem> {+ for await (const page of createPages()) {+ yield* extractItems(page);+ }+ }++ const result = {+ [Symbol.asyncIterator]() {+ return iterateAll();+ },+ then<TResult1 = TItem[], TResult2 = never>(+ onfulfilled?: ((value: TItem[]) => TResult1 | PromiseLike<TResult1>) | null,+ onrejected?: ((reason: any) => TResult2 | PromiseLike<TResult2>) | null,+ ): Promise<TResult1 | TResult2> {+ return getOrCreate().then(onfulfilled, onrejected);+ },+ catch<TResult = never>(+ onrejected?: ((reason: any) => TResult | PromiseLike<TResult>) | null,+ ): Promise<TItem[] | TResult> {+ return getOrCreate().catch(onrejected);+ },+ finally(onfinally?: (() => void) | null): Promise<TItem[]> {+ return getOrCreate().finally(onfinally);+ },+ [Symbol.toStringTag]: 'DualIterable',+ } as AsyncIterable<TItem> & Promise<TItem[]>;++ return result;+}+/*** Options for the static `open()` method on storage classes ({@apilink Dataset}, {@apilink KeyValueStore}, {@apilink RequestQueue}).*/diff --git a/packages/memory-storage/src/resource-clients/dataset.ts b/packages/memory-storage/src/resource-clients/dataset.tsindex 37a389d0f..a93c6a431 100644--- a/packages/memory-storage/src/resource-clients/dataset.ts+++ b/packages/memory-storage/src/resource-clients/dataset.ts@@ -1,18 +1,17 @@/* eslint-disable import/no-duplicates */import { randomUUID } from 'node:crypto';import { rm } from 'node:fs/promises';+import { resolve } from 'node:path';import type * as storage from '@crawlee/types';import type { Dictionary } from '@crawlee/types';import { s } from '@sapphire/shapeshift';-import { move } from 'fs-extra';import { scheduleBackgroundTask } from '../background-handler/index.js';-import { StorageTypes } from '../consts.js';import type { StorageImplementation } from '../fs/common.js';import { createDatasetStorageImplementation } from '../fs/dataset/index.js';import type { MemoryStorage } from '../index.js';-import { createPaginatedEntryList, createPaginatedList, resolveWithinDirectory } from '../utils.js';+import { resolveWithinDirectory } from '../utils.js';import { BaseClient } from './common/base-client.js';/**@@ -60,45 +59,7 @@ export class DatasetClient<Data extends Dictionary = Dictionary>return this.toDatasetInfo();}- async update(newFields: storage.DatasetClientUpdateOptions = {}): Promise<storage.DatasetInfo> {- const parsed = s- .object({- name: s.string().lengthGreaterThan(0).optional(),- })- .parse(newFields);-- // Skip if no changes- if (!parsed.name) {- return this.toDatasetInfo();- }-- // Check that name is not in use already- const existingStoreByName = this.client.datasetClientCache.find(- (store) => store.name?.toLowerCase() === parsed.name!.toLowerCase(),- );-- if (existingStoreByName) {- this.throwOnDuplicateEntry(StorageTypes.Dataset, 'name', parsed.name);- }-- this.name = parsed.name;-- const previousDir = this.datasetDirectory;-- this.datasetDirectory = resolveWithinDirectory(- this.client.datasetsDirectory,- parsed.name ?? this.name ?? this.id,- );-- await move(previousDir, this.datasetDirectory, { overwrite: true });-- // Update timestamps- this.updateTimestamps(true);-- return this.toDatasetInfo();- }-- async delete(): Promise<void> {+ async drop(): Promise<void> {const storeIndex = this.client.datasetClientCache.findIndex((store) => store.id === this.id);if (storeIndex !== -1) {@@ -110,35 +71,25 @@ export class DatasetClient<Data extends Dictionary = Dictionary>}}- async downloadItems(): Promise<Buffer> {- throw new Error('This method is not implemented in @crawlee/memory-storage');- }-- listItems(- options: storage.DatasetClientListOptions = {},- ): AsyncIterable<Data> & Promise<storage.PaginatedList<Data>> {- const { desc, limit, offset } = s- .object({- desc: s.boolean().optional(),- limit: s.number().int().optional(),- offset: s.number().int().optional(),- })- .parse(options);+ async purge(): Promise<void> {+ this.itemCount = 0;+ this.datasetEntries.clear();++ // Remove item files from disk but keep the directory+ if (this.client.persistStorage) {+ const { readdir } = await import('node:fs/promises');+ const entries = await readdir(this.datasetDirectory).catch(() => []);+ for (const entry of entries) {+ if (entry !== '__metadata__.json') {+ await rm(resolve(this.datasetDirectory, entry), { force: true });+ }+ }+ }- return createPaginatedList<Data>(- (pageOffset, pageLimit) =>- this.listItemsPage({- desc,- offset: pageOffset,- limit: Math.min(pageLimit, LIST_ITEMS_LIMIT),- }),- { offset, limit },- );+ this.updateTimestamps(true);}- listEntries(- options: storage.DatasetClientListOptions = {},- ): AsyncIterable<[number, Data]> & Promise<storage.PaginatedList<[number, Data]>> {+ getData(options: storage.DatasetClientListOptions = {}): Promise<storage.PaginatedList<Data>> {const { desc, limit, offset } = s.object({desc: s.boolean().optional(),@@ -147,18 +98,14 @@ export class DatasetClient<Data extends Dictionary = Dictionary>}).parse(options);- return createPaginatedEntryList<Data>(- (pageOffset, pageLimit) =>- this.listItemsPage({- desc,- offset: pageOffset,- limit: Math.min(pageLimit, LIST_ITEMS_LIMIT),- }),- { offset, limit },- );+ return this.getDataPage({+ desc,+ offset: offset ?? 0,+ limit: Math.min(limit ?? LIST_ITEMS_LIMIT, LIST_ITEMS_LIMIT),+ });}- private async listItemsPage(options: storage.DatasetClientListOptions = {}): Promise<storage.PaginatedList<Data>> {+ private async getDataPage(options: storage.DatasetClientListOptions = {}): Promise<storage.PaginatedList<Data>> {const { limit = LIST_ITEMS_LIMIT, offset = 0, desc } = options;const [start, end] = this.getStartAndEndIndexes(@@ -185,18 +132,8 @@ export class DatasetClient<Data extends Dictionary = Dictionary>};}- async pushItems(items: string | Data | string[] | Data[]): Promise<void> {- const rawItems = s- .union([- s.string(),- s.object<Data>({} as Data).passthrough(),- s.array(s.union([s.string(), s.object<Data>({} as Data).passthrough()])),- ])- .parse(items) as Data[];-- const normalized = this.normalizeItems(rawItems);-- for (const entry of normalized) {+ async pushData(items: Data[]): Promise<void> {+ for (const entry of items) {const idx = this.generateLocalEntryName(++this.itemCount);const storageEntry = createDatasetStorageImplementation({entityId: idx,@@ -233,39 +170,6 @@ export class DatasetClient<Data extends Dictionary = Dictionary>return [start, end] as const;}- /**- * To emulate API and split arrays of items into individual dataset items,- * we need to normalize the input items - which can be strings, objects- * or arrays of those - into objects, so that we can save them one by one- * later. We could potentially do this directly with strings, but let's- * not optimize prematurely.- */- private normalizeItems(items: string | Data | (string | Data)[]): Data[] {- if (typeof items === 'string') {- items = JSON.parse(items);- }-- return Array.isArray(items) ? items.map((item) => this.normalizeItem(item)) : [this.normalizeItem(items)];- }-- private normalizeItem(item: string | Data): Data {- if (typeof item === 'string') {- item = JSON.parse(item) as Data;- }-- if (Array.isArray(item)) {- throw new Error(- `Each dataset item can only be a single JSON object, not an array. Received: [${item.join(',\n')}]`,- );- }-- if (typeof item !== 'object' || item === null) {- throw new Error(`Each dataset item must be a JSON object. Received: ${item}`);- }-- return item;- }-private updateTimestamps(hasBeenModified: boolean) {this.accessedAt = new Date();diff --git a/packages/memory-storage/src/resource-clients/key-value-store.ts b/packages/memory-storage/src/resource-clients/key-value-store.tsindex 7f2406650..209035d4b 100644--- a/packages/memory-storage/src/resource-clients/key-value-store.ts+++ b/packages/memory-storage/src/resource-clients/key-value-store.ts@@ -1,31 +1,19 @@import { randomUUID } from 'node:crypto';import { rm } from 'node:fs/promises';-import { Readable } from 'node:stream';import type * as storage from '@crawlee/types';import { s } from '@sapphire/shapeshift';-import { move } from 'fs-extra';-import mime from 'mime-types';-import pLimit from 'p-limit';import { scheduleBackgroundTask } from '../background-handler/index.js';import { maybeParseBody } from '../body-parser.js';-import { DEFAULT_API_PARAM_LIMIT, StorageTypes } from '../consts.js';import type { StorageImplementation } from '../fs/common.js';import { createKeyValueStorageImplementation } from '../fs/key-value-store/index.js';import type { MemoryStorage } from '../index.js';-import {- createKeyList,- createKeyStringList,- createLazyIterablePromise,- isBuffer,- isStream,- resolveWithinDirectory,-} from '../utils.js';+import { isBuffer, isStream, resolveWithinDirectory } from '../utils.js';import { BaseClient } from './common/base-client.js';+import mime from 'mime-types';const DEFAULT_LOCAL_FILE_EXTENSION = 'bin';-const GET_RECORD_CONCURRENCY = 25;export interface KeyValueStoreClientOptions {name?: string;@@ -42,7 +30,7 @@ export interface InternalKeyRecord {filePath?: string;}-export class KeyValueStoreClient extends BaseClient {+export class KeyValueStoreClient extends BaseClient implements storage.KeyValueStoreClient {name?: string;createdAt = new Date();accessedAt = new Date();@@ -64,45 +52,7 @@ export class KeyValueStoreClient extends BaseClient {return this.toKeyValueStoreInfo();}- async update(newFields: storage.KeyValueStoreClientUpdateOptions = {}): Promise<storage.KeyValueStoreInfo> {- const parsed = s- .object({- name: s.string().lengthGreaterThan(0).optional(),- })- .parse(newFields);-- // Skip if no changes- if (!parsed.name) {- return this.toKeyValueStoreInfo();- }-- // Check that name is not in use already- const existingStoreByName = this.client.keyValueStoreCache.find(- (store) => store.name?.toLowerCase() === parsed.name!.toLowerCase(),- );-- if (existingStoreByName) {- this.throwOnDuplicateEntry(StorageTypes.KeyValueStore, 'name', parsed.name);- }-- this.name = parsed.name;-- const previousDir = this.keyValueStoreDirectory;-- this.keyValueStoreDirectory = resolveWithinDirectory(- this.client.keyValueStoresDirectory,- parsed.name ?? this.name ?? this.id,- );-- await move(previousDir, this.keyValueStoreDirectory, { overwrite: true });-- // Update timestamps- this.updateTimestamps(true);-- return this.toKeyValueStoreInfo();- }-- async delete(): Promise<void> {+ async drop(): Promise<void> {const storeIndex = this.client.keyValueStoreCache.findIndex((store) => store.id === this.id);if (storeIndex !== -1) {@@ -113,158 +63,27 @@ export class KeyValueStoreClient extends BaseClient {}}- listKeys(- options: storage.KeyValueStoreClientListOptions = {},- ): AsyncIterable<storage.KeyValueStoreItemData> & Promise<storage.KeyValueStoreClientListData> {- const { limit, exclusiveStartKey, prefix } = s- .object({- limit: s.number().greaterThan(0).optional(),- exclusiveStartKey: s.string().optional(),- collection: s.string().optional(), // This is ignored, but kept for validation consistency with API client.- prefix: s.string().optional(),- })- .parse(options);+ async purge(): Promise<void> {+ // Delete all entries+ const entriesToDelete = [...this.keyValueEntries.entries()];+ for (const [key, entry] of entriesToDelete) {+ this.keyValueEntries.delete(key);+ await entry.delete();+ }- return createKeyList(- (pageExclusiveStartKey) =>- this.listKeysPage({- limit: limit ?? DEFAULT_API_PARAM_LIMIT,- exclusiveStartKey: pageExclusiveStartKey,- prefix,- }),- { exclusiveStartKey, limit },- );+ this.updateTimestamps(true);}- keys(- options: storage.KeyValueStoreClientListOptions = {},- ): AsyncIterable<string> & Promise<storage.KeyValueStoreClientListData> {- const { limit, exclusiveStartKey, prefix } = s+ async listKeys(options: storage.KeyValueStoreListKeysOptions = {}): Promise<storage.KeyValueStoreItemData[]> {+ const { prefix, exclusiveStartKey, limit } = s.object({- limit: s.number().greaterThan(0).optional(),- exclusiveStartKey: s.string().optional(),- collection: s.string().optional(),prefix: s.string().optional(),+ exclusiveStartKey: s.string().optional(),+ limit: s.number().int().greaterThan(0).optional(),}).parse(options);- return createKeyStringList(- (pageExclusiveStartKey) =>- this.listKeysPage({- limit: limit ?? DEFAULT_API_PARAM_LIMIT,- exclusiveStartKey: pageExclusiveStartKey,- prefix,- }),- { exclusiveStartKey, limit },- );- }-- values(options: storage.KeyValueStoreClientListOptions = {}): AsyncIterable<unknown> & Promise<unknown[]> {- const keys = this.keys.bind(this);- const getRecord = this.getRecord.bind(this);- const limit = options.limit;-- const firstPageKeysPromise = keys(options);-- const getFirstPageValues = async () => {- const firstPageKeys = await firstPageKeysPromise;- const keysToFetch = limit !== undefined ? firstPageKeys.items.slice(0, limit) : firstPageKeys.items;- const limiter = pLimit(GET_RECORD_CONCURRENCY);- const results = await Promise.all(keysToFetch.map((item) => limiter(() => getRecord(item.key))));- return results.filter((r) => r !== undefined).map((r) => r.value);- };-- async function* asyncGenerator(): AsyncGenerator<unknown> {- const firstPageKeys = await firstPageKeysPromise;- let yielded = 0;-- for (const item of firstPageKeys.items) {- if (limit !== undefined && yielded >= limit) return;- const record = await getRecord(item.key);- if (record) {- yield record.value;- yielded++;- }- }-- if (firstPageKeys.nextExclusiveStartKey && (limit === undefined || yielded < limit)) {- for await (const key of keys({- ...options,- exclusiveStartKey: firstPageKeys.nextExclusiveStartKey,- })) {- if (limit !== undefined && yielded >= limit) return;- const record = await getRecord(key);- if (record) {- yield record.value;- yielded++;- }- }- }- }-- return createLazyIterablePromise(getFirstPageValues, asyncGenerator);- }-- entries(- options: storage.KeyValueStoreClientListOptions = {},- ): AsyncIterable<[string, unknown]> & Promise<[string, unknown][]> {- const keys = this.keys.bind(this);- const getRecord = this.getRecord.bind(this);- const limit = options.limit;-- const firstPageKeysPromise = keys(options);-- const getFirstPageEntries = async () => {- const firstPageKeys = await firstPageKeysPromise;- const keysToFetch = limit !== undefined ? firstPageKeys.items.slice(0, limit) : firstPageKeys.items;- const limiter = pLimit(GET_RECORD_CONCURRENCY);- const results = await Promise.all(- keysToFetch.map((item) =>- limiter(() => getRecord(item.key).then((record) => ({ key: item.key, record }))),- ),- );- return results- .filter((r) => r.record !== undefined)- .map((r) => [r.key, r.record!.value] as [string, unknown]);- };-- async function* asyncGenerator(): AsyncGenerator<[string, unknown]> {- const firstPageKeys = await firstPageKeysPromise;- let yielded = 0;-- for (const item of firstPageKeys.items) {- if (limit !== undefined && yielded >= limit) return;- const record = await getRecord(item.key);- if (record) {- yield [item.key, record.value];- yielded++;- }- }-- if (firstPageKeys.nextExclusiveStartKey && (limit === undefined || yielded < limit)) {- for await (const key of keys({- ...options,- exclusiveStartKey: firstPageKeys.nextExclusiveStartKey,- })) {- if (limit !== undefined && yielded >= limit) return;- const record = await getRecord(key);- if (record) {- yield [key, record.value];- yielded++;- }- }- }- }-- return createLazyIterablePromise(getFirstPageEntries, asyncGenerator);- }-- private async listKeysPage(- options: storage.KeyValueStoreClientListOptions = {},- ): Promise<storage.KeyValueStoreClientListData> {- const { limit = DEFAULT_API_PARAM_LIMIT, exclusiveStartKey, prefix } = options;-- const items = [];+ const items: storage.KeyValueStoreItemData[] = [];for (const storageEntry of this.keyValueEntries.values()) {const record = await storageEntry.get();@@ -277,36 +96,28 @@ export class KeyValueStoreClient extends BaseClient {}// Lexically sort to emulate API.- // TODO(vladfrangu): ensure the sorting works the same way as before (if it matters)- items.sort((a, b) => {- return a.key.localeCompare(b.key);- });+ items.sort((a, b) => a.key.localeCompare(b.key));- const filteredItems = items.filter((item) => !prefix || item.key.startsWith(prefix));+ let filteredItems = items.filter((item) => !prefix || item.key.startsWith(prefix));- let truncatedItems = filteredItems;if (exclusiveStartKey) {const keyPos = filteredItems.findIndex((item) => item.key === exclusiveStartKey);- if (keyPos !== -1) truncatedItems = filteredItems.slice(keyPos + 1);+ if (keyPos === -1) {+ throw new Error(+ `exclusiveStartKey "${exclusiveStartKey}" was not found in the key-value store. ` ++ `This is likely a bug — the key may have been deleted between paginated listKeys calls.`,+ );+ }+ filteredItems = filteredItems.slice(keyPos + 1);}- const limitedItems = truncatedItems.slice(0, limit);-- const lastItemInStore = filteredItems.at(-1);- const lastSelectedItem = limitedItems.at(-1);- const isLastSelectedItemAbsolutelyLast = lastItemInStore === lastSelectedItem;- const nextExclusiveStartKey = isLastSelectedItemAbsolutelyLast ? undefined : lastSelectedItem?.key;+ if (limit !== undefined) {+ filteredItems = filteredItems.slice(0, limit);+ }this.updateTimestamps(false);- return {- count: limitedItems.length,- limit,- exclusiveStartKey,- isTruncated: !isLastSelectedItemAbsolutelyLast,- nextExclusiveStartKey,- items: limitedItems,- };+ return filteredItems;}/**@@ -315,7 +126,7 @@ export class KeyValueStoreClient extends BaseClient {* Returns `undefined` if the record does not exist or has no associated file path (i.e., it is not stored as a file).* @param key The key of the record to generate the public URL for.*/- async getRecordPublicUrl(key: string): Promise<string | undefined> {+ async getPublicUrl(key: string): Promise<string | undefined> {s.string().parse(key);const storageEntry = await this.keyValueEntries.get(key)?.get();@@ -335,18 +146,8 @@ export class KeyValueStoreClient extends BaseClient {return this.keyValueEntries.has(key);}- async getRecord(- key: string,- options: storage.KeyValueStoreClientGetRecordOptions = {},- ): Promise<storage.KeyValueStoreRecord | undefined> {+ async getValue(key: string): Promise<storage.KeyValueStoreRecord | undefined> {s.string().parse(key);- s.object({- buffer: s.boolean().optional(),- // These options are ignored, but kept here- // for validation consistency with API client.- stream: s.boolean().optional(),- disableRedirect: s.boolean().optional(),- }).parse(options);const storageEntry = this.keyValueEntries.get(key);@@ -362,20 +163,15 @@ export class KeyValueStoreClient extends BaseClient {contentType: entry.contentType ?? (mime.contentType(entry.extension) as string),};- if (options.stream) {- record.value = Readable.from(record.value);- } else if (options.buffer) {- record.value = Buffer.from(record.value);- } else {- record.value = maybeParseBody(record.value, record.contentType!);- }+ // Auto-parse the body (JSON → object, text → string, etc.)+ record.value = maybeParseBody(record.value, record.contentType!);this.updateTimestamps(false);return record;}- async setRecord(record: storage.KeyValueStoreRecord): Promise<void> {+ async setValue(record: storage.KeyValueStoreRecord): Promise<void> {s.object({key: s.string().lengthGreaterThan(0),value: s.union([@@ -446,7 +242,7 @@ export class KeyValueStoreClient extends BaseClient {this.updateTimestamps(true);}- async deleteRecord(key: string): Promise<void> {+ async deleteValue(key: string): Promise<void> {s.string().parse(key);const entry = this.keyValueEntries.get(key);diff --git a/packages/memory-storage/src/resource-clients/request-queue.ts b/packages/memory-storage/src/resource-clients/request-queue.tsindex f60f4e805..80b85cde6 100644--- a/packages/memory-storage/src/resource-clients/request-queue.ts+++ b/packages/memory-storage/src/resource-clients/request-queue.ts@@ -4,10 +4,8 @@ import { rm } from 'node:fs/promises';import type * as storage from '@crawlee/types';import { AsyncQueue } from '@sapphire/async-queue';import { s } from '@sapphire/shapeshift';-import { move } from 'fs-extra/esm';import { scheduleBackgroundTask } from '../background-handler/index.js';-import { StorageTypes } from '../consts.js';import { createRequestQueueStorageImplementation } from '../fs/request-queue/index.js';import type { RequestQueueFileSystemEntry } from '../fs/request-queue/fs.js';import type { RequestQueueMemoryEntry } from '../fs/request-queue/memory.js';@@ -77,48 +75,7 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQuereturn this.toRequestQueueInfo();}- async update(newFields: { name?: string | undefined }): Promise<storage.RequestQueueInfo | undefined> {- // The validation is intentionally loose to prevent issues- // when swapping to a remote queue in production.- const parsed = s- .object({- name: s.string().lengthGreaterThan(0).optional(),- })- .passthrough()- .parse(newFields);-- // Skip if no changes- if (!parsed.name) {- return this.toRequestQueueInfo();- }-- // Check that name is not in use already- const existingQueueByName = this.client.requestQueueCache.find(- (queue) => queue.name?.toLowerCase() === parsed.name!.toLowerCase(),- );-- if (existingQueueByName) {- this.throwOnDuplicateEntry(StorageTypes.RequestQueue, 'name', parsed.name);- }-- this.name = parsed.name;-- const previousDir = this.requestQueueDirectory;-- this.requestQueueDirectory = resolveWithinDirectory(- this.client.requestQueuesDirectory,- parsed.name ?? this.name ?? this.id,- );-- await move(previousDir, this.requestQueueDirectory, { overwrite: true });-- // Update timestamps- this.updateTimestamps(true);-- return this.toRequestQueueInfo();- }-- async delete(): Promise<void> {+ async drop(): Promise<void> {const storeIndex = this.client.requestQueueCache.findIndex((queue) => queue.id === this.id);if (storeIndex !== -1) {@@ -130,6 +87,27 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQue}}+ async purge(): Promise<void> {+ // Clear all in-memory state+ this.requests.clear();+ this.forefrontRequestIds = [];+ this.handledRequestCount = 0;+ this.pendingRequestCount = 0;++ // Remove request files from disk but keep the directory+ if (this.client.persistStorage) {+ const { readdir } = await import('node:fs/promises');+ const entries = await readdir(this.requestQueueDirectory).catch(() => []);+ for (const entry of entries) {+ if (entry !== '__metadata__.json') {+ await rm(resolve(this.requestQueueDirectory, entry), { force: true });+ }+ }+ }++ this.updateTimestamps(true);+ }+private *requestKeyIterator(rqClient: RequestQueueClient): IterableIterator<string> {for (let i = this.forefrontRequestIds.length - 1; i >= 0; i--) {yield this.forefrontRequestIds[i];@@ -522,25 +500,6 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQue};}- async deleteRequest(id: string): Promise<void> {- const entry = this.requests.get(id);-- if (entry) {- const request = await entry.get();-- this.requests.delete(id);- this.updateTimestamps(true);-- if (request.orderNo) {- this.pendingRequestCount -= 1;- } else {- this.handledRequestCount -= 1;- }-- await entry.delete();- }- }-toRequestQueueInfo(): storage.RequestQueueInfo {return {accessedAt: this.accessedAt,diff --git a/packages/memory-storage/src/utils.ts b/packages/memory-storage/src/utils.tsindex 70a596195..eb96a77ba 100644--- a/packages/memory-storage/src/utils.ts+++ b/packages/memory-storage/src/utils.ts@@ -87,193 +87,3 @@ interface MetadataUpdate<Type extends EntityType, DataType> {writeMetadata: boolean;persistStorage: boolean;}--/**- * Creates a hybrid Promise + AsyncIterable for offset-based pagination (Dataset.listItems).- *- * The returned object can be:- * - Awaited directly to get the first page (backward compatible)- * - Used with `for await...of` to iterate through all items- */-export function createPaginatedList<Data>(- getPage: (offset: number, limit: number) => Promise<storage.PaginatedList<Data>>,- options: { offset?: number; limit?: number } = {},-): AsyncIterable<Data> & Promise<storage.PaginatedList<Data>> {- const offset = options.offset ?? 0;-- // Immediately fetch the first page (Infinity is used when no limit, gets clamped by Math.min downstream)- const firstPagePromise = getPage(offset, options.limit ?? Infinity);-- async function* asyncGenerator(): AsyncGenerator<Data> {- let currentPage = await firstPagePromise;- yield* currentPage.items;-- const limit = Math.min(options.limit ?? currentPage.total, currentPage.total);- let currentOffset = offset + currentPage.items.length;- let remainingItems = Math.min(currentPage.total - offset, limit) - currentPage.items.length;-- while (- currentPage.items.length > 0 && // Continue only if at least some items were returned in the last page.- remainingItems > 0- ) {- currentPage = await getPage(currentOffset, remainingItems);- yield* currentPage.items;- currentOffset += currentPage.items.length;- remainingItems -= currentPage.items.length;- }- }-- return Object.defineProperty(firstPagePromise, Symbol.asyncIterator, {- value: asyncGenerator,- }) as AsyncIterable<Data> & Promise<storage.PaginatedList<Data>>;-}--/**- * Creates a hybrid Promise + AsyncIterable for cursor-based pagination (KeyValueStore.listKeys).- *- * The returned object can be:- * - Awaited directly to get the first page (backward compatible)- * - Used with `for await...of` to iterate through all keys- */-export function createKeyList(- getPage: (exclusiveStartKey?: string) => Promise<storage.KeyValueStoreClientListData>,- options: { exclusiveStartKey?: string; limit?: number } = {},-): AsyncIterable<storage.KeyValueStoreItemData> & Promise<storage.KeyValueStoreClientListData> {- // Immediately fetch the first page- const firstPagePromise = getPage(options.exclusiveStartKey);-- async function* asyncGenerator(): AsyncGenerator<storage.KeyValueStoreItemData> {- let currentPage = await firstPagePromise;- yield* currentPage.items;-- let remainingItems = options.limit ? options.limit - currentPage.items.length : undefined;-- while (- currentPage.items.length > 0 &&- currentPage.nextExclusiveStartKey !== undefined &&- (remainingItems === undefined || remainingItems > 0)- ) {- currentPage = await getPage(currentPage.nextExclusiveStartKey);- yield* currentPage.items;- if (remainingItems !== undefined) {- remainingItems -= currentPage.items.length;- }- }- }-- return Object.defineProperty(firstPagePromise, Symbol.asyncIterator, {- value: asyncGenerator,- }) as AsyncIterable<storage.KeyValueStoreItemData> & Promise<storage.KeyValueStoreClientListData>;-}--/**- * Creates a hybrid Promise + AsyncIterable that yields only key strings (KeyValueStore.keys).- *- * The returned object can be:- * - Awaited directly to get the first page (backward compatible)- * - Used with `for await...of` to iterate through all key strings- */-export function createKeyStringList(- getPage: (exclusiveStartKey?: string) => Promise<storage.KeyValueStoreClientListData>,- options: { exclusiveStartKey?: string; limit?: number } = {},-): AsyncIterable<string> & Promise<storage.KeyValueStoreClientListData> {- // Immediately fetch the first page- const firstPagePromise = getPage(options.exclusiveStartKey);-- async function* asyncGenerator(): AsyncGenerator<string> {- let currentPage = await firstPagePromise;- for (const item of currentPage.items) {- yield item.key;- }-- let remainingItems = options.limit ? options.limit - currentPage.items.length : undefined;-- while (- currentPage.items.length > 0 &&- currentPage.nextExclusiveStartKey !== undefined &&- (remainingItems === undefined || remainingItems > 0)- ) {- currentPage = await getPage(currentPage.nextExclusiveStartKey);- for (const item of currentPage.items) {- yield item.key;- }- if (remainingItems !== undefined) {- remainingItems -= currentPage.items.length;- }- }- }-- return Object.defineProperty(firstPagePromise, Symbol.asyncIterator, {- value: asyncGenerator,- }) as AsyncIterable<string> & Promise<storage.KeyValueStoreClientListData>;-}--/**- * Creates a hybrid Promise + AsyncIterable for offset-based pagination with index-value entries (Dataset.listEntries).- *- * The returned object can be:- * - Awaited directly to get the first page with [index, item] tuples (backward compatible)- * - Used with `for await...of` to iterate through all entries as [index, item] tuples- */-export function createPaginatedEntryList<Data>(- getPage: (offset: number, limit: number) => Promise<storage.PaginatedList<Data>>,- options: { offset?: number; limit?: number } = {},-): AsyncIterable<[number, Data]> & Promise<storage.PaginatedList<[number, Data]>> {- const offset = options.offset ?? 0;-- // Immediately fetch the first page and transform items to entries- const firstPagePromise = getPage(offset, options.limit ?? Infinity).then((result) => ({- ...result,- items: result.items.map((item, i) => [offset + i, item] as [number, Data]),- }));-- async function* asyncGenerator(): AsyncGenerator<[number, Data]> {- let currentIndex = offset;- for await (const item of createPaginatedList(getPage, options)) {- yield [currentIndex++, item];- }- }-- return Object.defineProperty(firstPagePromise, Symbol.asyncIterator, {- value: asyncGenerator,- }) as AsyncIterable<[number, Data]> & Promise<storage.PaginatedList<[number, Data]>>;-}--/**- * Creates an object that acts as both a lazy Promise and an AsyncIterable.- * - When awaited, it triggers `promiseFactory` (bulk fetch, cached after first call).- * - When iterated with `for await...of`, it uses `iteratorFactory` (streaming, no bulk fetch).- */-export function createLazyIterablePromise<TPromise, TElement>(- promiseFactory: () => Promise<TPromise>,- iteratorFactory: () => AsyncGenerator<TElement>,-): AsyncIterable<TElement> & Promise<TPromise> {- let cached: Promise<TPromise> | null = null;- function getOrCreate(): Promise<TPromise> {- if (!cached) {- cached = promiseFactory();- }- return cached;- }-- const result = {- then<TResult1 = TPromise, TResult2 = never>(- onfulfilled?: ((value: TPromise) => TResult1 | PromiseLike<TResult1>) | null,- onrejected?: ((reason: any) => TResult2 | PromiseLike<TResult2>) | null,- ): Promise<TResult1 | TResult2> {- return getOrCreate().then(onfulfilled, onrejected);- },- catch<TResult = never>(- onrejected?: ((reason: any) => TResult | PromiseLike<TResult>) | null,- ): Promise<TPromise | TResult> {- return getOrCreate().catch(onrejected);- },- finally(onfinally?: (() => void) | null): Promise<TPromise> {- return getOrCreate().finally(onfinally);- },- [Symbol.asyncIterator]: iteratorFactory,- [Symbol.toStringTag]: 'Promise' as const,- };-- return result as AsyncIterable<TElement> & Promise<TPromise>;-}diff --git a/packages/memory-storage/test/async-iteration.test.ts b/packages/memory-storage/test/async-iteration.test.tsindex b3d9d7779..8fc6f96f6 100644--- a/packages/memory-storage/test/async-iteration.test.ts+++ b/packages/memory-storage/test/async-iteration.test.ts@@ -3,9 +3,6 @@ import path from 'node:path';import { MemoryStorage } from '@crawlee/memory-storage';import type { DatasetClient, KeyValueStoreClient } from '@crawlee/types';-import { vi } from 'vitest';--import { createLazyIterablePromise } from '../src/utils';describe('Async iteration support', () => {const localDataDirectory = path.resolve(__dirname, './tmp/async-iteration');@@ -18,17 +15,19 @@ describe('Async iteration support', () => {await rm(localDataDirectory, { force: true, recursive: true });});- describe('Dataset.listItems', () => {+ describe('Dataset.getData', () => {const elements = Array.from({ length: 25 }, (_, i) => ({ index: i }));- let dataset: DatasetClient;+ let dataset: DatasetClient<{ index: number }>;beforeAll(async () => {- dataset = await storage.createDatasetClient({ name: 'async-iteration-dataset' });- await dataset.pushItems(elements);+ dataset = (await storage.createDatasetClient({ name: 'async-iteration-dataset' })) as DatasetClient<{+ index: number;+ }>;+ await dataset.pushData(elements);});- test('can be awaited directly (backward compatibility)', async () => {- const result = await dataset.listItems({ limit: 10 });+ test('getData returns a paginated result', async () => {+ const result = await dataset.getData({ limit: 10 });expect(result.items).toHaveLength(10);expect(result.total).toBe(25);@@ -36,59 +35,32 @@ describe('Async iteration support', () => {expect(result.items).toStrictEqual(elements.slice(0, 10));});- test('can be used with for await...of to iterate all items', async () => {- const items: { index: number }[] = [];-- for await (const item of dataset.listItems()) {- items.push(item);- }-- expect(items).toHaveLength(25);- expect(items).toStrictEqual(elements);- });-- test('respects limit option when iterating', async () => {- const items: { index: number }[] = [];+ test('respects limit option', async () => {+ const result = await dataset.getData({ limit: 10 });- for await (const item of dataset.listItems({ limit: 10 })) {- items.push(item);- }-- expect(items).toHaveLength(10);- expect(items).toStrictEqual(elements.slice(0, 10));+ expect(result.items).toHaveLength(10);+ expect(result.items).toStrictEqual(elements.slice(0, 10));});- test('respects offset option when iterating', async () => {- const items: { index: number }[] = [];-- for await (const item of dataset.listItems({ offset: 5 })) {- items.push(item);- }+ test('respects offset option', async () => {+ const result = await dataset.getData({ offset: 5 });- expect(items).toHaveLength(20);- expect(items).toStrictEqual(elements.slice(5));+ expect(result.items).toHaveLength(20);+ expect(result.items).toStrictEqual(elements.slice(5));});- test('respects both offset and limit options when iterating', async () => {- const items: { index: number }[] = [];+ test('respects both offset and limit options', async () => {+ const result = await dataset.getData({ offset: 5, limit: 10 });- for await (const item of dataset.listItems({ offset: 5, limit: 10 })) {- items.push(item);- }-- expect(items).toHaveLength(10);- expect(items).toStrictEqual(elements.slice(5, 15));+ expect(result.items).toHaveLength(10);+ expect(result.items).toStrictEqual(elements.slice(5, 15));});- test('respects desc option when iterating', async () => {- const items: { index: number }[] = [];+ test('respects desc option', async () => {+ const result = await dataset.getData({ desc: true, limit: 5 });- for await (const item of dataset.listItems({ desc: true, limit: 5 })) {- items.push(item);- }-- expect(items).toHaveLength(5);- expect(items).toStrictEqual(elements.slice().reverse().slice(0, 5));+ expect(result.items).toHaveLength(5);+ expect(result.items).toStrictEqual(elements.slice().reverse().slice(0, 5));});});@@ -100,457 +72,44 @@ describe('Async iteration support', () => {kvStore = await storage.createKeyValueStoreClient({ name: 'async-iteration-kvs' });for (const key of keys) {- await kvStore.setRecord({ key, value: { data: key } });+ await kvStore.setValue({ key, value: { data: key } });}});- test('can be awaited directly (backward compatibility)', async () => {- const result = await kvStore.listKeys({ limit: 10 });-- expect(result.items).toHaveLength(10);- expect(result.isTruncated).toBe(true);- expect(result.items.map((i) => i.key)).toStrictEqual(keys.slice(0, 10));- });-- test('can be used with for await...of to iterate all keys', async () => {- const items: string[] = [];-- for await (const item of kvStore.listKeys()) {- items.push(item.key);- }+ test('returns all keys', async () => {+ const items = await kvStore.listKeys();expect(items).toHaveLength(25);- expect(items).toStrictEqual(keys);+ expect(items.map((i) => i.key)).toStrictEqual(keys);});- test('respects limit option when iterating (10 items, limit 2)', async () => {- // Create a fresh store with exactly 10 items to match the reported bug scenario- const testStore = await storage.createKeyValueStoreClient({ name: 'limit-test-kvs' });-- for (let i = 0; i < 10; i++) {- await testStore.setRecord({ key: `key-${i}`, value: `value-${i}` });- }-- const items: string[] = [];-- // This should only return 2 items, matching apify-client behavior- for await (const item of testStore.listKeys({ limit: 2 })) {- items.push(item.key);- }-- // Should only get 2 items, not all 10- expect(items).toHaveLength(2);- });-- test('respects exclusiveStartKey option when iterating', async () => {- const items: string[] = [];-- // Start after key-04 (index 4), should get keys 5-24- for await (const item of kvStore.listKeys({ exclusiveStartKey: 'key-04' })) {- items.push(item.key);- }-- expect(items).toHaveLength(20);- expect(items).toStrictEqual(keys.slice(5));- });-- test('respects prefix option when iterating', async () => {- const items: string[] = [];-+ test('respects prefix option', async () => {// Only keys starting with 'key-0' (key-00 to key-09)- for await (const item of kvStore.listKeys({ prefix: 'key-0' })) {- items.push(item.key);- }+ const items = await kvStore.listKeys({ prefix: 'key-0' });expect(items).toHaveLength(10);- expect(items).toStrictEqual(keys.slice(0, 10));+ expect(items.map((i) => i.key)).toStrictEqual(keys.slice(0, 10));});- });- describe('KeyValueStore.keys', () => {- const keys = Array.from({ length: 25 }, (_, i) => `key-${String(i).padStart(2, '0')}`);- let kvStore: KeyValueStoreClient;+ test('respects exclusiveStartKey option', async () => {+ const items = await kvStore.listKeys({ exclusiveStartKey: 'key-09' });- beforeAll(async () => {- kvStore = await storage.createKeyValueStoreClient({ name: 'async-iteration-kvs-keys' });-- for (const key of keys) {- await kvStore.setRecord({ key, value: { data: key } });- }+ expect(items).toHaveLength(15);+ expect(items.map((i) => i.key)).toStrictEqual(keys.slice(10));});- test('can be awaited directly (backward compatibility)', async () => {- const result = await kvStore.keys({ limit: 10 });-- // When awaited, returns the same structure as listKeys- expect(result.items).toHaveLength(10);- expect(result.isTruncated).toBe(true);- expect(result.items.map((i) => i.key)).toStrictEqual(keys.slice(0, 10));- });-- test('can be used with for await...of to iterate all keys as strings', async () => {- const items: string[] = [];-- for await (const key of kvStore.keys()) {- items.push(key);- }-- expect(items).toHaveLength(25);- expect(items).toStrictEqual(keys);- });-- test('yields strings directly, not objects', async () => {- for await (const key of kvStore.keys()) {- expect(typeof key).toBe('string');- break; // Only need to check the first one- }- });-- test('respects limit option when iterating', async () => {- const items: string[] = [];-- for await (const key of kvStore.keys({ limit: 10 })) {- items.push(key);- }-- expect(items).toHaveLength(10);- expect(items).toStrictEqual(keys.slice(0, 10));- });-- test('respects exclusiveStartKey option when iterating', async () => {- const items: string[] = [];-- // Start after key-04 (index 4), should get keys 5-24- for await (const key of kvStore.keys({ exclusiveStartKey: 'key-04' })) {- items.push(key);- }-- expect(items).toHaveLength(20);- expect(items).toStrictEqual(keys.slice(5));- });-- test('respects prefix option when iterating', async () => {- const items: string[] = [];-- // Only keys starting with 'key-0' (key-00 to key-09)- for await (const key of kvStore.keys({ prefix: 'key-0' })) {- items.push(key);- }-- expect(items).toHaveLength(10);- expect(items).toStrictEqual(keys.slice(0, 10));- });-- test('respects both exclusiveStartKey and limit options', async () => {- const items: string[] = [];-- for await (const key of kvStore.keys({ exclusiveStartKey: 'key-04', limit: 5 })) {- items.push(key);- }+ test('respects limit option', async () => {+ const items = await kvStore.listKeys({ limit: 5 });expect(items).toHaveLength(5);- expect(items).toStrictEqual(keys.slice(5, 10));- });- });-- describe('KeyValueStore.values', () => {- const keys = Array.from({ length: 25 }, (_, i) => `key-${String(i).padStart(2, '0')}`);- let kvStore: KeyValueStoreClient;-- beforeAll(async () => {- kvStore = await storage.createKeyValueStoreClient({ name: 'async-iteration-kvs-values' });-- for (const key of keys) {- await kvStore.setRecord({ key, value: { data: key } });- }- });-- test('can be awaited directly (backward compatibility)', async () => {- const values = await kvStore.values({ limit: 10 });-- expect(values).toHaveLength(10);- expect(Array.isArray(values)).toBe(true);- expect(values[0]).toStrictEqual({ data: 'key-00' });- });-- test('can be used with for await...of to iterate all values', async () => {- const values: unknown[] = [];-- for await (const value of kvStore.values()) {- values.push(value);- }-- expect(values).toHaveLength(25);- expect(values.every((v) => v && typeof v === 'object')).toBe(true);- });-- test('yields values directly, not KeyValueStoreRecord objects', async () => {- for await (const value of kvStore.values()) {- // Should be the actual value, not a record wrapper- expect(value).toStrictEqual({ data: 'key-00' });- expect(value).not.toHaveProperty('contentType');- break; // Only need to check the first one- }- });-- test('respects limit option when iterating', async () => {- const values: unknown[] = [];-- for await (const value of kvStore.values({ limit: 10 })) {- values.push(value);- }-- expect(values).toHaveLength(10);- });-- test('respects exclusiveStartKey option when iterating', async () => {- const values: unknown[] = [];-- // Start after key-04 (index 4), should get keys 5-24- for await (const value of kvStore.values({ exclusiveStartKey: 'key-04' })) {- values.push(value);- }-- expect(values).toHaveLength(20);- });-- test('respects prefix option when iterating', async () => {- const values: unknown[] = [];-- // Only keys starting with 'key-0' (key-00 to key-09)- for await (const value of kvStore.values({ prefix: 'key-0' })) {- values.push(value);- }-- expect(values).toHaveLength(10);- });-- test('fetches actual record values', async () => {- const values: unknown[] = [];-- for await (const value of kvStore.values({ limit: 3 })) {- values.push(value);- }-- expect(values[0]).toStrictEqual({ data: 'key-00' });- expect(values[1]).toStrictEqual({ data: 'key-01' });- expect(values[2]).toStrictEqual({ data: 'key-02' });- });- });-- describe('KeyValueStore.entries', () => {- const keys = Array.from({ length: 25 }, (_, i) => `key-${String(i).padStart(2, '0')}`);- let kvStore: KeyValueStoreClient;-- beforeAll(async () => {- kvStore = await storage.createKeyValueStoreClient({ name: 'async-iteration-kvs-entries' });-- for (const key of keys) {- await kvStore.setRecord({ key, value: { data: key } });- }- });-- test('can be awaited directly (backward compatibility)', async () => {- const entries = await kvStore.entries({ limit: 10 });-- expect(entries).toHaveLength(10);- expect(Array.isArray(entries)).toBe(true);- // Each entry is a [key, value] tuple- expect(entries[0][0]).toBe('key-00');- expect(entries[0][1]).toStrictEqual({ data: 'key-00' });+ expect(items.map((i) => i.key)).toStrictEqual(keys.slice(0, 5));});- test('can be used with for await...of to iterate all entries', async () => {- const entries: [string, unknown][] = [];-- for await (const entry of kvStore.entries()) {- entries.push(entry);- }-- expect(entries).toHaveLength(25);- expect(entries.map(([key]) => key)).toStrictEqual(keys);- });-- test('yields [key, value] tuples', async () => {- for await (const [key, value] of kvStore.entries()) {- expect(typeof key).toBe('string');- expect(key).toBe('key-00');- expect(value).toStrictEqual({ data: 'key-00' });- // Value should not be a record wrapper- expect(value).not.toHaveProperty('contentType');- break; // Only need to check the first one- }- });-- test('respects limit option when iterating', async () => {- const entries: [string, unknown][] = [];-- for await (const entry of kvStore.entries({ limit: 10 })) {- entries.push(entry);- }-- expect(entries).toHaveLength(10);- expect(entries.map(([key]) => key)).toStrictEqual(keys.slice(0, 10));- });-- test('respects exclusiveStartKey option when iterating', async () => {- const entries: [string, unknown][] = [];-- // Start after key-04 (index 4), should get keys 5-24- for await (const entry of kvStore.entries({ exclusiveStartKey: 'key-04' })) {- entries.push(entry);- }-- expect(entries).toHaveLength(20);- expect(entries.map(([key]) => key)).toStrictEqual(keys.slice(5));- });-- test('respects prefix option when iterating', async () => {- const entries: [string, unknown][] = [];-- // Only keys starting with 'key-0' (key-00 to key-09)- for await (const entry of kvStore.entries({ prefix: 'key-0' })) {- entries.push(entry);- }-- expect(entries).toHaveLength(10);- expect(entries.map(([key]) => key)).toStrictEqual(keys.slice(0, 10));- });-- test('values in entries match expected data', async () => {- for await (const [key, value] of kvStore.entries({ limit: 5 })) {- expect(value).toStrictEqual({ data: key });- }- });- });+ test('respects exclusiveStartKey and limit together', async () => {+ const items = await kvStore.listKeys({ exclusiveStartKey: 'key-04', limit: 5 });- describe('createLazyIterablePromise', () => {- test('promise factory is not called until awaited', async () => {- const promiseFactory = vi.fn(() => Promise.resolve([1, 2, 3]));- async function* iteratorFactory() {- yield 1;- yield 2;- yield 3;- }-- const result = createLazyIterablePromise<number[], number>(promiseFactory, iteratorFactory);-- // Factory should not be called yet- expect(promiseFactory).not.toHaveBeenCalled();-- // Now await it- const values = await result;- expect(promiseFactory).toHaveBeenCalledTimes(1);- expect(values).toStrictEqual([1, 2, 3]);- });-- test('iterating does not trigger the promise factory', async () => {- const promiseFactory = vi.fn(() => Promise.resolve([1, 2, 3]));- async function* iteratorFactory() {- yield 10;- yield 20;- yield 30;- }-- const result = createLazyIterablePromise<number[], number>(promiseFactory, iteratorFactory);-- const items: number[] = [];- for await (const item of result) {- items.push(item);- }-- expect(items).toStrictEqual([10, 20, 30]);- expect(promiseFactory).not.toHaveBeenCalled();- });-- test('promise factory result is cached across multiple awaits', async () => {- const promiseFactory = vi.fn(() => Promise.resolve([1, 2, 3]));- async function* iteratorFactory() {- yield 1;- }-- const result = createLazyIterablePromise<number[], number>(promiseFactory, iteratorFactory);-- await result;- await result;- await result;-- expect(promiseFactory).toHaveBeenCalledTimes(1);- });- });-- describe('KeyValueStore.values lazy promise behavior', () => {- let kvStore: KeyValueStoreClient;-- beforeAll(async () => {- kvStore = await storage.createKeyValueStoreClient({ name: 'lazy-test-kvs-values' });-- for (let i = 0; i < 5; i++) {- await kvStore.setRecord({ key: `key-${i}`, value: { data: i } });- }- });-- test('calling values() does not immediately fetch records', async () => {- const getRecordSpy = vi.spyOn(kvStore, 'getRecord');-- // Call values() but do not await or iterate- const result = kvStore.values();-- // getRecord should not have been called yet (lazy)- // Note: keys may be fetched eagerly, but record values should not- // We need to wait a tick to ensure no async work triggered getRecord- await new Promise((resolve) => setTimeout(resolve, 50));- expect(getRecordSpy).not.toHaveBeenCalled();-- // Clean up: consume the result to avoid dangling promises- await result;- getRecordSpy.mockRestore();- });-- test('iterating and awaiting produce the same values', async () => {- const awaited = await kvStore.values();-- const iterated: unknown[] = [];- for await (const value of kvStore.values()) {- iterated.push(value);- }-- expect(awaited).toStrictEqual(iterated);- });- });-- describe('KeyValueStore.entries lazy promise behavior', () => {- let kvStore: KeyValueStoreClient;-- beforeAll(async () => {- kvStore = await storage.createKeyValueStoreClient({ name: 'lazy-test-kvs-entries' });-- for (let i = 0; i < 5; i++) {- await kvStore.setRecord({ key: `key-${i}`, value: { data: i } });- }- });-- test('calling entries() does not immediately fetch records', async () => {- const getRecordSpy = vi.spyOn(kvStore, 'getRecord');-- const result = kvStore.entries();-- await new Promise((resolve) => setTimeout(resolve, 50));- expect(getRecordSpy).not.toHaveBeenCalled();-- await result;- getRecordSpy.mockRestore();- });-- test('iterating and awaiting produce the same entries', async () => {- const awaited = await kvStore.entries();-- const iterated: [string, unknown][] = [];- for await (const entry of kvStore.entries()) {- iterated.push(entry);- }-- expect(awaited).toStrictEqual(iterated);+ expect(items).toHaveLength(5);+ expect(items.map((i) => i.key)).toStrictEqual(keys.slice(5, 10));});});});diff --git a/packages/memory-storage/test/fs-fallback.test.ts b/packages/memory-storage/test/fs-fallback.test.tsindex ca8ac5ff7..21c4f6434 100644--- a/packages/memory-storage/test/fs-fallback.test.ts+++ b/packages/memory-storage/test/fs-fallback.test.ts@@ -61,7 +61,7 @@ describe('fallback to fs for reading', () => {expect(defaultStoreInfo.name).toEqual('default');expect(defaultStoreInfo.createdAt).toEqual(expectedFsDate);- const input = await defaultStore.getRecord('INPUT');+ const input = await defaultStore.getValue('INPUT');expect(input).toStrictEqual<KeyValueStoreRecord>({key: 'INPUT',value: { foo: 'bar but from fs' },@@ -72,7 +72,7 @@ describe('fallback to fs for reading', () => {test('attempting to read "other" key value store with no "__metadata__" present should read from fs, even if accessed without generating id first', async () => {const otherStore = await storage.createKeyValueStoreClient({ name: 'other' });- const input = await otherStore.getRecord('INPUT');+ const input = await otherStore.getValue('INPUT');expect(input).toStrictEqual<KeyValueStoreRecord>({key: 'INPUT',value: { foo: 'bar but from fs' },@@ -89,7 +89,7 @@ describe('fallback to fs for reading', () => {test('attempting to read "no-ext" key value store should load the missing extension file correctly', async () => {const noExtStore = await storage.createKeyValueStoreClient({ name: 'no-ext' });- const input = await noExtStore.getRecord('INPUT');+ const input = await noExtStore.getValue('INPUT');expect(input).toStrictEqual<KeyValueStoreRecord>({key: 'INPUT',value: JSON.stringify({ foo: 'bar but from fs' }),@@ -100,7 +100,7 @@ describe('fallback to fs for reading', () => {test('attempting to read "invalid-json" key value store should ignore the invalid "INPUT" json file', async () => {const invalidJsonStore = await storage.createKeyValueStoreClient({ name: 'invalid-json' });- const input = await invalidJsonStore.getRecord('INPUT');+ const input = await invalidJsonStore.getValue('INPUT');expect(input).toBeUndefined();});});diff --git a/packages/memory-storage/test/key-value-store/stream.test.ts b/packages/memory-storage/test/key-value-store/stream.test.tsindex 8577a4266..bfbd05cd3 100644--- a/packages/memory-storage/test/key-value-store/stream.test.ts+++ b/packages/memory-storage/test/key-value-store/stream.test.ts@@ -12,11 +12,11 @@ describe('KeyValueStore should drain streams when setting records', () => {test('should drain stream', async () => {const defaultStore = await storage.createKeyValueStoreClient({ name: 'default' });- await defaultStore.setRecord({ key: 'streamz', value: fsStream, contentType: 'text/plain' });+ await defaultStore.setValue({ key: 'streamz', value: fsStream, contentType: 'text/plain' });expect(fsStream.destroyed).toBeTruthy();- const record = await defaultStore.getRecord('streamz');+ const record = await defaultStore.getValue('streamz');expect(record!.value.toString('utf8')).toEqual('helloworld');});});diff --git a/packages/memory-storage/test/no-crash-on-big-buffers.test.ts b/packages/memory-storage/test/no-crash-on-big-buffers.test.tsindex 473bc0b3f..136960b14 100644--- a/packages/memory-storage/test/no-crash-on-big-buffers.test.ts+++ b/packages/memory-storage/test/no-crash-on-big-buffers.test.ts@@ -36,7 +36,7 @@ describe('MemoryStorage should not crash when saving a big buffer', () => {}try {- await store.setRecord({ key: 'owo.zip', value: zip });+ await store.setValue({ key: 'owo.zip', value: zip });} catch (err) {expect(err).not.toBeDefined();}diff --git a/packages/memory-storage/test/no-writing-to-disk.test.ts b/packages/memory-storage/test/no-writing-to-disk.test.tsindex b7cc86f52..9006957a2 100644--- a/packages/memory-storage/test/no-writing-to-disk.test.ts+++ b/packages/memory-storage/test/no-writing-to-disk.test.ts@@ -21,7 +21,7 @@ describe('persistStorage option', () => {test('creating a key-value pair in a key-value store should not write data to the disk', async () => {const keyValueStore = await storage.createKeyValueStoreClient();- await keyValueStore.setRecord({ key: 'foo', value: 'test' });+ await keyValueStore.setValue({ key: 'foo', value: 'test' });// We check that reading the directory for the store throws an error, which means it wasn't created on diskawait expect(async () => readdir(localDataDirectory)).rejects.toThrow();@@ -38,7 +38,7 @@ describe('persistStorage option', () => {test('creating a key-value pair in a key-value store should not write data to the disk, but it should write the __metadata__ file', async () => {const keyValueStore = await storage.createKeyValueStoreClient();- await keyValueStore.setRecord({ key: 'foo', value: 'test' });+ await keyValueStore.setValue({ key: 'foo', value: 'test' });const keyValueStoreInfo = await keyValueStore.getMetadata();const storePath = resolve(storage.keyValueStoresDirectory, keyValueStoreInfo.id);diff --git a/packages/memory-storage/test/request-queue/forefront.test.ts b/packages/memory-storage/test/request-queue/forefront.test.tsindex 45553b599..7cb02e608 100644--- a/packages/memory-storage/test/request-queue/forefront.test.ts+++ b/packages/memory-storage/test/request-queue/forefront.test.ts@@ -15,7 +15,7 @@ describe('RequestQueueV1 respects `forefront` in `listHead`', () => {});afterEach(async () => {- await requestQueue.delete();+ await requestQueue.drop();});test('requests without `forefront` respect sequential order', async () => {@@ -202,7 +202,7 @@ describe('RequestQueueV2 respects `forefront` in `listAndLockHead`', () => {});afterEach(async () => {- await requestQueue.delete();+ await requestQueue.drop();});test('requests without `forefront` respect sequential order', async () => {diff --git a/packages/memory-storage/test/request-queue/handledRequestCount-should-update.test.ts b/packages/memory-storage/test/request-queue/handledRequestCount-should-update.test.tsindex c711408ab..fd3987f7a 100644--- a/packages/memory-storage/test/request-queue/handledRequestCount-should-update.test.ts+++ b/packages/memory-storage/test/request-queue/handledRequestCount-should-update.test.ts@@ -37,19 +37,6 @@ describe('RequestQueue handledRequestCount should update', () => {expect(updatedStatistics.handledRequestCount).toEqual(2);});- test('deleting a request should decrement the handledRequestCount', async () => {- const { requestId } = await requestQueue.addRequest({- url: 'http://example.com/3',- uniqueKey: '3',- handledAt: new Date().toISOString(),- });-- await requestQueue.deleteRequest(requestId);-- const updatedStatistics = await requestQueue.getMetadata();- expect(updatedStatistics.handledRequestCount).toEqual(2);- });-test('updating an already handled request should not increment the handledRequestCount again', async () => {const { requestId } = await requestQueue.addRequest({url: 'http://example.com/4',@@ -57,7 +44,7 @@ describe('RequestQueue handledRequestCount should update', () => {handledAt: new Date().toISOString(),});- const { handledRequestCount } = (await requestQueue.get())!;+ const { handledRequestCount } = await requestQueue.getMetadata();await requestQueue.updateRequest({url: 'http://example.com/4',@@ -66,7 +53,7 @@ describe('RequestQueue handledRequestCount should update', () => {handledAt: new Date().toISOString(),});- const updatedStatistics = await requestQueue.get();- expect(updatedStatistics?.handledRequestCount).toEqual(handledRequestCount);+ const updatedStatistics = await requestQueue.getMetadata();+ expect(updatedStatistics.handledRequestCount).toEqual(handledRequestCount);});});diff --git a/packages/memory-storage/test/reverse-datataset-list.test.ts b/packages/memory-storage/test/reverse-datataset-list.test.tsindex d28553795..28b0388c8 100644--- a/packages/memory-storage/test/reverse-datataset-list.test.ts+++ b/packages/memory-storage/test/reverse-datataset-list.test.ts@@ -6,7 +6,7 @@ import type { DatasetClient } from '@crawlee/types';const elements = Array.from({ length: 10 }, (_, i) => ({ number: i }));-describe('Dataset#listItems respects the desc option', () => {+describe('Dataset#getData respects the desc option', () => {const localDataDirectory = resolve(import.meta.dirname, './tmp/desc');const storage = new MemoryStorage({localDataDirectory,@@ -22,32 +22,32 @@ describe('Dataset#listItems respects the desc option', () => {beforeAll(async () => {dataset = await storage.createDatasetClient({ name: 'false' });- await dataset.pushItems(elements);+ await dataset.pushData(elements);});test('with desc: false', async () => {- const result = await dataset.listItems({ desc: false, limit: 5 });+ const result = await dataset.getData({ desc: false, limit: 5 });expect(result.items).toHaveLength(5);expect(result.items).toStrictEqual(elements.slice(0, 5));});test('with desc: true', async () => {- const result = await dataset.listItems({ desc: true, limit: 5 });+ const result = await dataset.getData({ desc: true, limit: 5 });expect(result.items).toHaveLength(5);expect(result.items).toStrictEqual(elements.slice().reverse().slice(0, 5));});test('with desc: false and offset: 2', async () => {- const result = await dataset.listItems({ desc: false, limit: 5, offset: 2 });+ const result = await dataset.getData({ desc: false, limit: 5, offset: 2 });expect(result.items).toHaveLength(5);expect(result.items).toStrictEqual(elements.slice(2, 7));});test('with desc: true and offset: 2', async () => {- const result = await dataset.listItems({ desc: true, limit: 5, offset: 2 });+ const result = await dataset.getData({ desc: true, limit: 5, offset: 2 });expect(result.items).toHaveLength(5);expect(result.items).toStrictEqual(elements.slice().reverse().slice(2, 7));diff --git a/packages/memory-storage/test/write-metadata.test.ts b/packages/memory-storage/test/write-metadata.test.tsindex 68a465a90..80a6686ac 100644--- a/packages/memory-storage/test/write-metadata.test.ts+++ b/packages/memory-storage/test/write-metadata.test.ts@@ -30,7 +30,7 @@ describe('writeMetadata option', () => {test('creating a key-value pair in a key-value store should not write __metadata__.json file for the value', async () => {const keyValueStore = await storage.createKeyValueStoreClient();- await keyValueStore.setRecord({ key: 'foo', value: 'test' });+ await keyValueStore.setValue({ key: 'foo', value: 'test' });const keyValueStoreInfo = await keyValueStore.getMetadata();const expectedFilePath = resolve(storage.keyValueStoresDirectory, `${keyValueStoreInfo.id}/foo.txt`);@@ -62,7 +62,7 @@ describe('writeMetadata option', () => {test('creating a key-value pair in a key-value store should write __metadata__.json file for the value', async () => {const keyValueStore = await storage.createKeyValueStoreClient();- await keyValueStore.setRecord({ key: 'foo', value: 'test' });+ await keyValueStore.setValue({ key: 'foo', value: 'test' });const keyValueStoreInfo = await keyValueStore.getMetadata();const expectedFilePath = resolve(storage.keyValueStoresDirectory, `${keyValueStoreInfo.id}/foo.txt`);diff --git a/packages/types/src/storages.ts b/packages/types/src/storages.tsindex f0d11453c..b992a5817 100644--- a/packages/types/src/storages.ts+++ b/packages/types/src/storages.ts@@ -30,10 +30,6 @@ export interface PaginatedList<Data> {items: Data[];}-export interface DatasetClientUpdateOptions {- name?: string;-}-export interface DatasetClientListOptions {desc?: boolean;limit?: number;@@ -66,14 +62,18 @@ export interface DatasetClient<Data extends Dictionary = Dictionary> {* for a storage that has been removed.*/getMetadata(): Promise<DatasetInfo>;- update(newFields: DatasetClientUpdateOptions): Promise<Partial<DatasetInfo>>;- delete(): Promise<void>;- downloadItems(...args: unknown[]): Promise<Buffer>;- listItems(options?: DatasetClientListOptions): AsyncIterable<Data> & Promise<PaginatedList<Data>>;- listEntries?(- options?: DatasetClientListOptions,- ): AsyncIterable<[number, Data]> & Promise<PaginatedList<[number, Data]>>;- pushItems(items: Data | Data[] | string | string[]): Promise<void>;++ /** Remove the dataset and all its data. */+ drop(): Promise<void>;++ /** Remove all items from the dataset but keep the dataset itself. */+ purge(): Promise<void>;++ /** Add items to the dataset. */+ pushData(items: Data[]): Promise<void>;++ /** Fetch a page of items from the dataset. */+ getData(options?: DatasetClientListOptions): Promise<PaginatedList<Data>>;}export interface KeyValueStoreStats {@@ -102,20 +102,13 @@ export interface KeyValueStoreRecord {contentType?: string;}-export interface KeyValueStoreRecordOptions {- timeoutSecs?: number;- doNotRetryTimeouts?: boolean;-}--export interface KeyValueStoreClientUpdateOptions {- name?: string;-}--export interface KeyValueStoreClientListOptions {- limit?: number;- exclusiveStartKey?: string;- collection?: string;+export interface KeyValueStoreListKeysOptions {+ /** If set, only keys that start with this prefix are returned. */prefix?: string;+ /** All keys up to this one are skipped from the result. */+ exclusiveStartKey?: string;+ /** Maximum number of keys to return. */+ limit?: number;}export interface KeyValueStoreItemData {@@ -123,20 +116,6 @@ export interface KeyValueStoreItemData {size: number;}-export interface KeyValueStoreClientListData {- count: number;- limit: number;- exclusiveStartKey?: string;- isTruncated: boolean;- nextExclusiveStartKey?: string;- items: KeyValueStoreItemData[];-}--export interface KeyValueStoreClientGetRecordOptions {- buffer?: boolean;- stream?: boolean;-}-/*** Key-value Store client.*/@@ -149,19 +128,30 @@ export interface KeyValueStoreClient {* for a storage that has been removed.*/getMetadata(): Promise<KeyValueStoreInfo>;- update(newFields: KeyValueStoreClientUpdateOptions): Promise<Partial<KeyValueStoreInfo>>;- delete(): Promise<void>;- listKeys(- options?: KeyValueStoreClientListOptions,- ): Partial<AsyncIterable<KeyValueStoreItemData>> & Promise<KeyValueStoreClientListData>;- keys?(options?: KeyValueStoreClientListOptions): AsyncIterable<string> & Promise<KeyValueStoreClientListData>;- values?(options?: KeyValueStoreClientListOptions): AsyncIterable<unknown> & Promise<unknown[]>;- entries?(options?: KeyValueStoreClientListOptions): AsyncIterable<[string, unknown]> & Promise<[string, unknown][]>;++ /** Remove the key-value store and all its data. */+ drop(): Promise<void>;++ /** Remove all records from the store but keep the store itself. */+ purge(): Promise<void>;++ /** Get a record value by key. Returns the parsed value or `undefined` if not found. */+ getValue(key: string): Promise<KeyValueStoreRecord | undefined>;++ /** Set a record value. */+ setValue(record: KeyValueStoreRecord): Promise<void>;++ /** Delete a record by key. */+ deleteValue(key: string): Promise<void>;++ /** List keys in the store. Returns at most `limit` keys starting after `exclusiveStartKey`. */+ listKeys(options?: KeyValueStoreListKeysOptions): Promise<KeyValueStoreItemData[]>;++ /** Get the public URL for a record, or `undefined` if unavailable. */+ getPublicUrl(key: string): Promise<string | undefined>;++ /** Check whether a record with the given key exists. */recordExists(key: string): Promise<boolean>;- getRecordPublicUrl(key: string): Promise<string | undefined>;- getRecord(key: string, options?: KeyValueStoreClientGetRecordOptions): Promise<KeyValueStoreRecord | undefined>;- setRecord(record: KeyValueStoreRecord, options?: KeyValueStoreRecordOptions): Promise<void>;- deleteRecord(key: string): Promise<void>;}export interface RequestQueueStats {@@ -284,14 +274,18 @@ export interface RequestQueueClient {* for a storage that has been removed.*/getMetadata(): Promise<RequestQueueInfo>;- update(newFields: { name?: string }): Promise<Partial<RequestQueueInfo> | undefined>;- delete(): Promise<void>;++ /** Remove the request queue and all its data. */+ drop(): Promise<void>;++ /** Remove all requests from the queue but keep the queue itself. */+ purge(): Promise<void>;+listHead(options?: ListOptions): Promise<QueueHead>;addRequest(request: RequestSchema, options?: RequestOptions): Promise<QueueOperationInfo>;batchAddRequests(requests: RequestSchema[], options?: RequestOptions): Promise<BatchAddRequestsResult>;getRequest(id: string): Promise<RequestOptions | undefined>;updateRequest(request: UpdateRequestSchema, options?: RequestOptions): Promise<QueueOperationInfo>;- deleteRequest(id: string): Promise<unknown>;listAndLockHead(options: ListAndLockOptions): Promise<ListAndLockHeadResult>;prolongRequestLock(id: string, options: ProlongRequestLockOptions): Promise<ProlongRequestLockResult>;deleteRequestLock(id: string, options?: DeleteRequestLockOptions): Promise<void>;diff --git a/test/core/crawlers/adaptive_playwright_crawler.test.ts b/test/core/crawlers/adaptive_playwright_crawler.test.tsindex b17c2ca0b..e01344e0b 100644--- a/test/core/crawlers/adaptive_playwright_crawler.test.ts+++ b/test/core/crawlers/adaptive_playwright_crawler.test.ts@@ -526,9 +526,9 @@ describe('AdaptivePlaywrightCrawler', () => {await crawler.run();const store = await localStorageEmulator.getKeyValueStore();- expect((await store.getRecord('1'))!.value).toEqual({ content: 42 });- expect((await store.getRecord('2'))!.value).toEqual({ content: 42 });- expect((await store.getRecord('3'))!.value).toEqual({ content: 42 });+ expect((await store.getValue('1'))!.value).toEqual({ content: 42 });+ expect((await store.getValue('2'))!.value).toEqual({ content: 42 });+ expect((await store.getValue('3'))!.value).toEqual({ content: 42 });});test('should not allow direct key-value store manipulation', async () => {@@ -562,7 +562,7 @@ describe('AdaptivePlaywrightCrawler', () => {);const store = await localStorageEmulator.getKeyValueStore();- expect(await store.getRecord('1')).toBeUndefined();+ expect(await store.getValue('1')).toBeUndefined();});test('should persist RenderingTypePredictor state on PERSIST_STATE events', async () => {diff --git a/test/core/crawlers/rendering_type_predictor.test.ts b/test/core/crawlers/rendering_type_predictor.test.tsindex ac44fc089..9abedb8a5 100644--- a/test/core/crawlers/rendering_type_predictor.test.ts+++ b/test/core/crawlers/rendering_type_predictor.test.ts@@ -39,7 +39,7 @@ describe('RenderingTypePredictor', () => {await predictor['state'].persistState(); // Access private state for persistence// Verify state was persisted- const persistedRecord = await store.getRecord(persistStateKey);+ const persistedRecord = await store.getValue(persistStateKey);expect(persistedRecord).not.toBeNull();expect(persistedRecord?.value).toBeDefined();diff --git a/test/core/crawlers/statistics.test.ts b/test/core/crawlers/statistics.test.tsindex 56c56e0b6..9f4535939 100644--- a/test/core/crawlers/statistics.test.ts+++ b/test/core/crawlers/statistics.test.ts@@ -189,10 +189,6 @@ describe('Statistics', () => {// @ts-expect-error Accessing private propstats.persistStateKey,{ ...state, ...rest },- {- doNotRetryTimeouts: true,- timeoutSecs: 30,- },);}, 2000);diff --git a/test/core/recoverable_state.test.ts b/test/core/recoverable_state.test.tsindex 947ea7bff..c5a659d3e 100644--- a/test/core/recoverable_state.test.ts+++ b/test/core/recoverable_state.test.ts@@ -180,7 +180,7 @@ describe('RecoverableState', () => {await recoverableState.persistState();const persistedState = JSON.parse(- (await (await localStorageEmulator.getKeyValueStore()).getRecord('test-key'))?.value,+ (await (await localStorageEmulator.getKeyValueStore()).getValue('test-key'))?.value,);expect(persistedState).toMatchObject({data: { value: 'updated' },diff --git a/test/core/storages/dataset.test.ts b/test/core/storages/dataset.test.tsindex c4021cf9b..51b5e29ed 100644--- a/test/core/storages/dataset.test.ts+++ b/test/core/storages/dataset.test.ts@@ -1,4 +1,4 @@-import { checkAndSerialize, chunkBySize, Configuration, Dataset, KeyValueStore, serviceLocator } from '@crawlee/core';+import { assertJsonSerializable, Dataset, KeyValueStore, serviceLocator } from '@crawlee/core';import type { Dictionary } from '@crawlee/utils';import { MemoryStorageEmulator } from '../../shared/MemoryStorageEmulator.js';@@ -30,28 +30,28 @@ describe('dataset', () => {test('should work', async () => {const dataset = await createDataset();- const pushItemSpy = vitest.spyOn(dataset.client, 'pushItems');+ const pushDataSpy = vitest.spyOn(dataset.client, 'pushData');- const mockPushItems = pushItemSpy.mockResolvedValueOnce(undefined);+ const mockPushData = pushDataSpy.mockResolvedValueOnce(undefined);await dataset.pushData({ foo: 'bar' });- expect(mockPushItems).toBeCalledTimes(1);- expect(mockPushItems).toBeCalledWith(JSON.stringify({ foo: 'bar' }));+ expect(mockPushData).toHaveBeenCalledTimes(1);+ expect(mockPushData).toHaveBeenCalledWith([{ foo: 'bar' }]);- const mockPushItems2 = pushItemSpy.mockResolvedValueOnce(undefined);+ const mockPushData2 = pushDataSpy.mockResolvedValueOnce(undefined);await dataset.pushData([{ foo: 'hotel;' }, { foo: 'restaurant' }]);- expect(mockPushItems2).toBeCalledTimes(2);- expect(mockPushItems2).toBeCalledWith(JSON.stringify([{ foo: 'hotel;' }, { foo: 'restaurant' }]));+ expect(mockPushData2).toHaveBeenCalledTimes(2);+ expect(mockPushData2).toHaveBeenCalledWith([{ foo: 'hotel;' }, { foo: 'restaurant' }]);- const mockDelete = vitest.spyOn(dataset.client, 'delete').mockResolvedValueOnce(undefined);+ const mockDrop = vitest.spyOn(dataset.client, 'drop').mockResolvedValueOnce(undefined);await dataset.drop();- expect(mockDelete).toBeCalledTimes(1);- expect(mockDelete).toHaveBeenLastCalledWith();+ expect(mockDrop).toHaveBeenCalledTimes(1);+ expect(mockDrop).toHaveBeenLastCalledWith();});test('should successfully save large data', async () => {@@ -59,15 +59,13 @@ describe('dataset', () => {const dataset = await createDataset();- const mockPushItems = vitest.spyOn(dataset.client, 'pushItems');- mockPushItems.mockResolvedValueOnce(undefined);- mockPushItems.mockResolvedValueOnce(undefined);+ const mockPushData = vitest.spyOn(dataset.client, 'pushData');+ mockPushData.mockResolvedValueOnce(undefined);await dataset.pushData([{ foo: half }, { bar: half }]);- expect(mockPushItems).toBeCalledTimes(2);- expect(mockPushItems).toHaveBeenNthCalledWith(1, JSON.stringify([{ foo: half }]));- expect(mockPushItems).toHaveBeenNthCalledWith(2, JSON.stringify([{ bar: half }]));+ expect(mockPushData).toHaveBeenCalledTimes(1);+ expect(mockPushData).toHaveBeenCalledWith([{ foo: half }, { bar: half }]);});test('should successfully save lots of small data', async () => {@@ -75,43 +73,16 @@ describe('dataset', () => {const string = mockData(MAX_PAYLOAD_SIZE_BYTES / count);const chunk = { foo: string, bar: 'baz' };const data = Array(count).fill(chunk);- const expectedFirst = JSON.stringify(Array(count - 1).fill(chunk));- const expectedSecond = JSON.stringify([chunk]);const dataset = await createDataset();- const mockPushItems = vitest.spyOn(dataset.client, 'pushItems');- mockPushItems.mockResolvedValueOnce(undefined);- mockPushItems.mockResolvedValueOnce(undefined);+ const mockPushData = vitest.spyOn(dataset.client, 'pushData');+ mockPushData.mockResolvedValueOnce(undefined);await dataset.pushData(data);- expect(mockPushItems).toBeCalledTimes(2);- expect(mockPushItems).toHaveBeenNthCalledWith(1, expectedFirst);- expect(mockPushItems).toHaveBeenNthCalledWith(2, expectedSecond);- });-- test('should throw on too large file', async () => {- const full = mockData(MAX_PAYLOAD_SIZE_BYTES);- const dataset = await createDataset();- try {- await dataset.pushData({ foo: full });- throw new Error('Should fail!');- } catch (err) {- expect(err).toBeInstanceOf(Error);- expect((err as Error).message).toMatch('Data item is too large');- }- });- test('should throw on too large file in an array', async () => {- const full = mockData(MAX_PAYLOAD_SIZE_BYTES);- const dataset = await createDataset();- try {- await dataset.pushData([{ foo: 0 }, { foo: 1 }, { foo: 2 }, { foo: full }, { foo: 4 }]);- throw new Error('Should fail!');- } catch (err) {- expect(err).toBeInstanceOf(Error);- expect((err as Error).message).toMatch('Data item at index 3 is too large');- }+ expect(mockPushData).toHaveBeenCalledTimes(1);+ expect(mockPushData).toHaveBeenCalledWith(data);});test('getData() should work', async () => {@@ -126,27 +97,22 @@ describe('dataset', () => {desc: false,};- const mockListItems = vitest.spyOn(dataset.client, 'listItems');- mockListItems.mockResolvedValueOnce(expected);+ const mockGetData = vitest.spyOn(dataset.client, 'getData');+ mockGetData.mockResolvedValueOnce(expected);const result = await dataset.getData({ limit: 2, offset: 3 });- expect(mockListItems).toHaveBeenLastCalledWith({+ expect(mockGetData).toHaveBeenLastCalledWith({limit: 2,offset: 3,});expect(result).toEqual(expected);- let e;- const spy = vitest.spyOn(dataset.client, 'listItems').mockImplementation(() => {++ vitest.spyOn(dataset.client, 'getData').mockImplementation(() => {throw new Error('Cannot create a string longer than 0x3fffffe7 characters');});- try {- await dataset.getData();- } catch (err) {- e = err;- }- expect((e as Error).message).toEqual(+ await expect(dataset.getData()).rejects.toThrow('dataset.getData(): The response is too large for parsing. You can fix this by lowering the "limit" option.',);});@@ -192,17 +158,17 @@ describe('dataset', () => {desc: false,};- const mockListItems = vitest.spyOn(dataset.client, 'listItems');- mockListItems.mockResolvedValueOnce(firstResolve);- mockListItems.mockResolvedValueOnce(secondResolve);+ const mockGetData = vitest.spyOn(dataset.client, 'getData');+ mockGetData.mockResolvedValueOnce(firstResolve);+ mockGetData.mockResolvedValueOnce(secondResolve);const restoreAndVerify = () => {- expect(mockListItems).toBeCalledTimes(2);- expect(mockListItems).toHaveBeenNthCalledWith(1, {+ expect(mockGetData).toHaveBeenCalledTimes(2);+ expect(mockGetData).toHaveBeenNthCalledWith(1, {limit: 2,offset: 0,});- expect(mockListItems).toHaveBeenNthCalledWith(2, {+ expect(mockGetData).toHaveBeenNthCalledWith(2, {limit: 2,offset: 2,});@@ -331,8 +297,8 @@ describe('dataset', () => {test('reduce() uses first value as memo if no memo is provided', async () => {const dataset = await createDataset('some-id', 'some-name');- const mockListItems = vitest.spyOn(dataset.client, 'listItems');- mockListItems.mockResolvedValueOnce({+ const mockGetData = vitest.spyOn(dataset.client, 'getData');+ mockGetData.mockResolvedValueOnce({items: [{ foo: 4 }, { foo: 5 }],limit: 2,total: 4,@@ -340,7 +306,7 @@ describe('dataset', () => {count: 2,desc: false,});- mockListItems.mockResolvedValueOnce({+ mockGetData.mockResolvedValueOnce({items: [{ foo: 4 }, { foo: 1 }],limit: 2,total: 4,@@ -362,12 +328,12 @@ describe('dataset', () => {},);- expect(mockListItems).toBeCalledTimes(2);- expect(mockListItems).toHaveBeenNthCalledWith(1, {+ expect(mockGetData).toHaveBeenCalledTimes(2);+ expect(mockGetData).toHaveBeenNthCalledWith(1, {limit: 2,offset: 0,});- expect(mockListItems).toHaveBeenNthCalledWith(2, {+ expect(mockGetData).toHaveBeenNthCalledWith(2, {limit: 2,offset: 2,});@@ -401,7 +367,7 @@ describe('dataset', () => {'Expected `data` to be of type `object` but received type `boolean`',);await expect(dataset.pushData(() => {})).rejects.toThrow(- 'Data item is not an object. You can push only objects into a dataset.',+ 'Data item at index 0 is not an object. You can push only objects into a dataset.',);const circularObj = {} as Dictionary;@@ -412,59 +378,22 @@ describe('dataset', () => {});describe('utils', () => {- test('checkAndSerialize() works', () => {- // Basic- const obj = { foo: 'bar' };- const json = JSON.stringify(obj);- expect(checkAndSerialize({}, 100)).toBe('{}');- expect(checkAndSerialize(obj, 100)).toEqual(json);- // With index- expect(checkAndSerialize(obj, 100, 1)).toEqual(json);- // Too large- expect(() => checkAndSerialize(obj, 5)).toThrowError(Error);- expect(() => checkAndSerialize(obj, 5, 7)).toThrowError(Error);- // Bad JSON+ test('assertJsonSerializable() works', () => {+ // Valid objects+ expect(() => assertJsonSerializable({})).not.toThrow();+ expect(() => assertJsonSerializable({ foo: 'bar' })).not.toThrow();+ expect(() => assertJsonSerializable({ foo: 'bar' }, 1)).not.toThrow();+ // Circular referenceconst bad = {} as Dictionary;bad.bad = bad;- expect(() => checkAndSerialize(bad, 100)).toThrowError(Error);- // Bad data- const str = 'hello';- expect(() => checkAndSerialize(str, 100)).toThrowError(Error);- expect(() => checkAndSerialize([], 100)).toThrowError(Error);- expect(() => checkAndSerialize([str, str], 100)).toThrowError(Error);+ expect(() => assertJsonSerializable(bad)).toThrow('not serializable to JSON');+ // Non-objects+ expect(() => assertJsonSerializable('hello')).toThrow('not an object');+ expect(() => assertJsonSerializable([])).toThrow('not an object');+ expect(() => assertJsonSerializable(['a', 'b'])).toThrow('not an object');+ // With index in error message+ expect(() => assertJsonSerializable('hello', 3)).toThrow('at index 3');});- test('chunkBySize', () => {- const obj = { foo: 'bar' };- const json = JSON.stringify(obj);- const size = Buffer.byteLength(json);- const triple = [json, json, json];- const originalTriple = [obj, obj, obj];- const chunk = `[${json}]`;- const tripleChunk = `[${json},${json},${json}]`;- const tripleSize = Buffer.byteLength(tripleChunk);- // Empty array- expect(chunkBySize([], 10)).toEqual([]);- // Fits easily- expect(chunkBySize([json], size + 10)).toEqual([json]);- expect(chunkBySize(triple, tripleSize + 10)).toEqual([tripleChunk]);- // Parses back to original objects- expect(originalTriple).toEqual(JSON.parse(tripleChunk));- // Fits exactly- expect(chunkBySize([json], size)).toEqual([json]);- expect(chunkBySize(triple, tripleSize)).toEqual([tripleChunk]);- // Chunks large items individually- expect(chunkBySize(triple, size)).toEqual(triple);- expect(chunkBySize(triple, size + 1)).toEqual(triple);- expect(chunkBySize(triple, size + 2)).toEqual([chunk, chunk, chunk]);- // Chunks smaller items together- expect(chunkBySize(triple, 2 * size + 3)).toEqual([`[${json},${json}]`, chunk]);- expect(chunkBySize([...triple, ...triple], 2 * size + 3)).toEqual([- `[${json},${json}]`,- `[${json},${json}]`,- `[${json},${json}]`,- ]);- });-describe('exportToJSON', () => {const dataToPush = [{@@ -574,39 +503,30 @@ describe('dataset', () => {expect(items).toEqual(testData);});- test('values() can be awaited directly (hybrid usage)', async () => {+ test('values() respects limit when iterating', async () => {const dataset = await Dataset.open();await dataset.pushData(testData);- const result = await dataset.values();-- expect(result.items).toEqual(testData);- expect(result.total).toBe(3);- expect(result.count).toBe(3);- expect(result.offset).toBe(0);- });-- test('values() respects limit when awaited directly', async () => {- const dataset = await Dataset.open();- await dataset.pushData(testData);-- const result = await dataset.values({ limit: 2 });+ const items = [];+ for await (const item of dataset.values({ limit: 2 })) {+ items.push(item);+ }- expect(result.items).toHaveLength(2);- expect(result.items).toEqual(testData.slice(0, 2));- expect(result.total).toBe(3);- expect(result.count).toBe(2);+ expect(items).toHaveLength(2);+ expect(items).toEqual(testData.slice(0, 2));});- test('values() respects offset when awaited directly', async () => {+ test('values() respects offset when iterating', async () => {const dataset = await Dataset.open();await dataset.pushData(testData);- const result = await dataset.values({ offset: 1 });+ const items = [];+ for await (const item of dataset.values({ offset: 1 })) {+ items.push(item);+ }- expect(result.items).toHaveLength(2);- expect(result.items).toEqual(testData.slice(1));- expect(result.offset).toBe(1);+ expect(items).toHaveLength(2);+ expect(items).toEqual(testData.slice(1));});test('entries() should iterate over index-item pairs', async () => {@@ -640,49 +560,36 @@ describe('dataset', () => {]);});- test('entries() can be awaited directly (hybrid usage)', async () => {+ test('entries() respects limit when iterating', async () => {const dataset = await Dataset.open();await dataset.pushData(testData);- const result = await dataset.entries();-- expect(result.items).toEqual([- [0, { id: 1, name: 'Alice' }],- [1, { id: 2, name: 'Bob' }],- [2, { id: 3, name: 'Charlie' }],- ]);- expect(result.total).toBe(3);- expect(result.count).toBe(3);- expect(result.offset).toBe(0);- });-- test('entries() respects limit when awaited directly', async () => {- const dataset = await Dataset.open();- await dataset.pushData(testData);-- const result = await dataset.entries({ limit: 2 });+ const entries = [];+ for await (const entry of dataset.entries({ limit: 2 })) {+ entries.push(entry);+ }- expect(result.items).toHaveLength(2);- expect(result.items).toEqual([+ expect(entries).toHaveLength(2);+ expect(entries).toEqual([[0, { id: 1, name: 'Alice' }],[1, { id: 2, name: 'Bob' }],]);- expect(result.total).toBe(3);- expect(result.count).toBe(2);});- test('entries() respects offset when awaited directly', async () => {+ test('entries() respects offset when iterating', async () => {const dataset = await Dataset.open();await dataset.pushData(testData);- const result = await dataset.entries({ offset: 1 });+ const entries = [];+ for await (const entry of dataset.entries({ offset: 1 })) {+ entries.push(entry);+ }- expect(result.items).toHaveLength(2);- expect(result.items).toEqual([+ expect(entries).toHaveLength(2);+ expect(entries).toEqual([[1, { id: 2, name: 'Bob' }],[2, { id: 3, name: 'Charlie' }],]);- expect(result.offset).toBe(1);});test('Symbol.asyncIterator should iterate over items', async () => {@@ -707,6 +614,68 @@ describe('dataset', () => {expect(items).toEqual([]);});++ test('await values() should return all items as a flat array', async () => {+ const dataset = await Dataset.open();+ await dataset.pushData(testData);++ const items = await dataset.values();++ expect(items).toEqual(testData);+ });++ test('await values() should respect limit', async () => {+ const dataset = await Dataset.open();+ await dataset.pushData(testData);++ const items = await dataset.values({ limit: 2 });++ expect(items).toHaveLength(2);+ expect(items).toEqual(testData.slice(0, 2));+ });++ test('await values() should respect offset', async () => {+ const dataset = await Dataset.open();+ await dataset.pushData(testData);++ const items = await dataset.values({ offset: 1 });++ expect(items).toHaveLength(2);+ expect(items).toEqual(testData.slice(1));+ });++ test('await entries() should return all entries as a flat array', async () => {+ const dataset = await Dataset.open();+ await dataset.pushData(testData);++ const entries = await dataset.entries();++ expect(entries).toEqual([+ [0, { id: 1, name: 'Alice' }],+ [1, { id: 2, name: 'Bob' }],+ [2, { id: 3, name: 'Charlie' }],+ ]);+ });++ test('await entries() should respect offset', async () => {+ const dataset = await Dataset.open();+ await dataset.pushData(testData);++ const entries = await dataset.entries({ offset: 1 });++ expect(entries).toEqual([+ [1, { id: 2, name: 'Bob' }],+ [2, { id: 3, name: 'Charlie' }],+ ]);+ });++ test('await on empty dataset should return empty array', async () => {+ const dataset = await Dataset.open();++ const items = await dataset.values();++ expect(items).toEqual([]);+ });});});});diff --git a/test/core/storages/key_value_store.test.ts b/test/core/storages/key_value_store.test.tsindex d28d39b54..41f0ed767 100644--- a/test/core/storages/key_value_store.test.ts+++ b/test/core/storages/key_value_store.test.ts@@ -1,6 +1,6 @@import { PassThrough } from 'node:stream';-import { Configuration, KeyValueStore, maybeStringify, serviceLocator } from '@crawlee/core';+import { KeyValueStore, maybeStringify, serviceLocator } from '@crawlee/core';import type { Dictionary } from '@crawlee/utils';import { MemoryStorageEmulator } from '../../shared/MemoryStorageEmulator.js';@@ -32,30 +32,24 @@ describe('KeyValueStore', () => {const recordStr = JSON.stringify(record, null, 2);// Set record- const mockSetRecord = vitest+ const mockSetValue = vitest// @ts-expect-error Accessing private property- .spyOn(store.client, 'setRecord')+ .spyOn(store.client, 'setValue').mockResolvedValueOnce(undefined);await store.setValue('key-1', record);- expect(mockSetRecord).toBeCalledTimes(1);- expect(mockSetRecord).toBeCalledWith(- {- key: 'key-1',- value: recordStr,- contentType: 'application/json; charset=utf-8',- },- {- doNotRetryTimeouts: undefined,- timeoutSecs: undefined,- },- );+ expect(mockSetValue).toHaveBeenCalledTimes(1);+ expect(mockSetValue).toHaveBeenCalledWith({+ key: 'key-1',+ value: recordStr,+ contentType: 'application/json; charset=utf-8',+ });// Get Record- const mockGetRecord = vitest+ const mockGetValue = vitest// @ts-expect-error Accessing private property- .spyOn(store.client, 'getRecord')+ .spyOn(store.client, 'getValue').mockResolvedValueOnce({key: 'key-1',value: record,@@ -64,8 +58,8 @@ describe('KeyValueStore', () => {const response = await store.getValue('key-1');- expect(mockGetRecord).toBeCalledTimes(1);- expect(mockGetRecord).toBeCalledWith('key-1');+ expect(mockGetValue).toHaveBeenCalledTimes(1);+ expect(mockGetValue).toHaveBeenCalledWith('key-1');expect(response).toEqual(record);// Record Exists@@ -76,31 +70,31 @@ describe('KeyValueStore', () => {const exists = await store.recordExists('key-1');- expect(mockRecordExists).toBeCalledTimes(1);- expect(mockRecordExists).toBeCalledWith('key-1');+ expect(mockRecordExists).toHaveBeenCalledTimes(1);+ expect(mockRecordExists).toHaveBeenCalledWith('key-1');expect(exists).toBe(true);// Delete Record- const mockDeleteRecord = vitest+ const mockDeleteValue = vitest// @ts-expect-error Accessing private property- .spyOn(store.client, 'deleteRecord')+ .spyOn(store.client, 'deleteValue').mockResolvedValueOnce(undefined);await store.setValue('key-1', null);- expect(mockDeleteRecord).toBeCalledTimes(1);- expect(mockDeleteRecord).toBeCalledWith('key-1');+ expect(mockDeleteValue).toHaveBeenCalledTimes(1);+ expect(mockDeleteValue).toHaveBeenCalledWith('key-1');// Drop store- const mockDelete = vitest+ const mockDrop = vitest// @ts-expect-error Accessing private property- .spyOn(store.client, 'delete')+ .spyOn(store.client, 'drop').mockResolvedValueOnce(undefined);await store.drop();- expect(mockDelete).toBeCalledTimes(1);- expect(mockDelete).toHaveBeenLastCalledWith();+ expect(mockDrop).toHaveBeenCalledTimes(1);+ expect(mockDrop).toHaveBeenLastCalledWith();});describe('getValue', () => {@@ -127,13 +121,13 @@ describe('KeyValueStore', () => {getValueSpy.mockImplementationOnce(async () => 123);const val = await KeyValueStore.getValue('key-1');- expect(getValueSpy).toBeCalledTimes(1);- expect(getValueSpy).toBeCalledWith('key-1', undefined);+ expect(getValueSpy).toHaveBeenCalledTimes(1);+ expect(getValueSpy).toHaveBeenCalledWith('key-1', undefined);expect(val).toBe(123);const val2 = await KeyValueStore.getValue('key-2', 321);- expect(getValueSpy).toBeCalledTimes(2);- expect(getValueSpy).toBeCalledWith('key-2', 321);+ expect(getValueSpy).toHaveBeenCalledTimes(2);+ expect(getValueSpy).toHaveBeenCalledWith('key-2', 321);expect(val2).toBe(321);});});@@ -162,8 +156,8 @@ describe('KeyValueStore', () => {recordExistsSpy.mockImplementationOnce(async () => false);const val = await KeyValueStore.recordExists('key-1');- expect(recordExistsSpy).toBeCalledTimes(1);- expect(recordExistsSpy).toBeCalledWith('key-1');+ expect(recordExistsSpy).toHaveBeenCalledTimes(1);+ expect(recordExistsSpy).toHaveBeenCalledWith('key-1');expect(val).toBe(false);});});@@ -264,25 +258,19 @@ describe('KeyValueStore', () => {test('correctly adds charset to content type', async () => {const store = await createKeyValueStore('my-store-id-1');- const mockSetRecord = vitest+ const mockSetValue = vitest// @ts-expect-error Accessing private property- .spyOn(store.client, 'setRecord')+ .spyOn(store.client, 'setValue').mockResolvedValueOnce(undefined);await store.setValue('key-1', 'xxxx', { contentType: 'text/plain; charset=utf-8' });- expect(mockSetRecord).toBeCalledTimes(1);- expect(mockSetRecord).toBeCalledWith(- {- key: 'key-1',- value: 'xxxx',- contentType: 'text/plain; charset=utf-8',- },- {- doNotRetryTimeouts: undefined,- timeoutSecs: undefined,- },- );+ expect(mockSetValue).toHaveBeenCalledTimes(1);+ expect(mockSetValue).toHaveBeenCalledWith({+ key: 'key-1',+ value: 'xxxx',+ contentType: 'text/plain; charset=utf-8',+ });});test('correctly passes object values as JSON', async () => {@@ -291,112 +279,64 @@ describe('KeyValueStore', () => {const record = { foo: 'bar' };const recordStr = JSON.stringify(record, null, 2);- const mockSetRecord = vitest+ const mockSetValue = vitest// @ts-expect-error Accessing private property- .spyOn(store.client, 'setRecord')+ .spyOn(store.client, 'setValue').mockResolvedValueOnce(undefined);await store.setValue('key-1', record);- expect(mockSetRecord).toBeCalledTimes(1);- expect(mockSetRecord).toBeCalledWith(- {- key: 'key-1',- value: recordStr,- contentType: 'application/json; charset=utf-8',- },- {- doNotRetryTimeouts: undefined,- timeoutSecs: undefined,- },- );- });-- test('correctly passes timeout options', async () => {- const store = await createKeyValueStore('my-store-id-1');-- const record = { foo: 'bar' };- const recordStr = JSON.stringify(record, null, 2);-- const mockSetRecord = vitest- // @ts-expect-error Accessing private property- .spyOn(store.client, 'setRecord')- .mockResolvedValueOnce(undefined);-- await store.setValue('key-1', record, {- timeoutSecs: 1,- doNotRetryTimeouts: true,+ expect(mockSetValue).toHaveBeenCalledTimes(1);+ expect(mockSetValue).toHaveBeenCalledWith({+ key: 'key-1',+ value: recordStr,+ contentType: 'application/json; charset=utf-8',});-- expect(mockSetRecord).toBeCalledTimes(1);- expect(mockSetRecord).toBeCalledWith(- {- key: 'key-1',- value: recordStr,- contentType: 'application/json; charset=utf-8',- },- {- doNotRetryTimeouts: true,- timeoutSecs: 1,- },- );});test('correctly passes raw string values', async () => {const store = await createKeyValueStore('my-store-id-1');- const mockSetRecord = vitest+ const mockSetValue = vitest// @ts-expect-error Accessing private property- .spyOn(store.client, 'setRecord')+ .spyOn(store.client, 'setValue').mockResolvedValueOnce(undefined);await store.setValue('key-1', 'xxxx', { contentType: 'text/plain; charset=utf-8' });- expect(mockSetRecord).toBeCalledTimes(1);- expect(mockSetRecord).toBeCalledWith(- {- key: 'key-1',- value: 'xxxx',- contentType: 'text/plain; charset=utf-8',- },- {- doNotRetryTimeouts: undefined,- timeoutSecs: undefined,- },- );+ expect(mockSetValue).toHaveBeenCalledTimes(1);+ expect(mockSetValue).toHaveBeenCalledWith({+ key: 'key-1',+ value: 'xxxx',+ contentType: 'text/plain; charset=utf-8',+ });});test('correctly passes raw Buffer values', async () => {const store = await createKeyValueStore('my-store-id-1');- const mockSetRecord = vitest+ const mockSetValue = vitest// @ts-expect-error Accessing private property- .spyOn(store.client, 'setRecord')+ .spyOn(store.client, 'setValue').mockResolvedValueOnce(undefined);const value = Buffer.from('some text value');await store.setValue('key-1', value, { contentType: 'image/jpeg; charset=something' });- expect(mockSetRecord).toBeCalledTimes(1);- expect(mockSetRecord).toBeCalledWith(- {- key: 'key-1',- value,- contentType: 'image/jpeg; charset=something',- },- {- doNotRetryTimeouts: undefined,- timeoutSecs: undefined,- },- );+ expect(mockSetValue).toHaveBeenCalledTimes(1);+ expect(mockSetValue).toHaveBeenCalledWith({+ key: 'key-1',+ value,+ contentType: 'image/jpeg; charset=something',+ });});test('correctly passes a stream', async () => {const store = await createKeyValueStore('my-store-id-1');- const mockSetRecord = vitest+ const mockSetValue = vitest// @ts-expect-error Accessing private property- .spyOn(store.client, 'setRecord')+ .spyOn(store.client, 'setValue').mockResolvedValueOnce(undefined);const value = new PassThrough();@@ -405,18 +345,12 @@ describe('KeyValueStore', () => {value.end();value.destroy();- expect(mockSetRecord).toHaveBeenCalledTimes(1);- expect(mockSetRecord).toHaveBeenCalledWith(- {- key: 'key-1',- value,- contentType: 'plain/text',- },- {- doNotRetryTimeouts: undefined,- timeoutSecs: undefined,- },- );+ expect(mockSetValue).toHaveBeenCalledTimes(1);+ expect(mockSetValue).toHaveBeenCalledWith({+ key: 'key-1',+ value,+ contentType: 'plain/text',+ });});});@@ -447,7 +381,7 @@ describe('KeyValueStore', () => {const obj = {} as Dictionary;obj.self = obj;- expect(() => maybeStringify(obj, { contentType: null as any })).toThrowError(+ expect(() => maybeStringify(obj, { contentType: null as any })).toThrow('The "value" parameter cannot be stringified to JSON: Converting circular structure to JSON',);});@@ -517,51 +451,23 @@ describe('KeyValueStore', () => {// @ts-expect-error Accessing private propertyconst mockListKeys = vitest.spyOn(store.client, 'listKeys');- mockListKeys.mockResolvedValueOnce({- isTruncated: true,- exclusiveStartKey: 'key0',- nextExclusiveStartKey: 'key2',- items: [- { key: 'key1', size: 1 },- { key: 'key2', size: 2 },- ],- count: 2,- limit: 2,- });-- mockListKeys.mockResolvedValueOnce({- isTruncated: true,- exclusiveStartKey: 'key0',- nextExclusiveStartKey: 'key4',- items: [- { key: 'key3', size: 3 },- { key: 'key4', size: 4 },- ],- count: 1,- limit: 2,- });-- mockListKeys.mockResolvedValueOnce({- isTruncated: false,- exclusiveStartKey: 'key0',- nextExclusiveStartKey: undefined,- items: [{ key: 'key5', size: 5 }],- count: 1,- limit: 1,- });+ mockListKeys.mockResolvedValueOnce([+ { key: 'key1', size: 1 },+ { key: 'key2', size: 2 },+ { key: 'key3', size: 3 },+ { key: 'key4', size: 4 },+ { key: 'key5', size: 5 },+ ]);const results: [string, number, { size: number }][] = [];await store.forEachKey(async (key, index, info) => {results.push([key, index, info]);},- { exclusiveStartKey: 'key0', prefix: 'img/' },+ { prefix: 'img/' },);- expect(mockListKeys).toBeCalledTimes(3);- expect(mockListKeys).toHaveBeenNthCalledWith(1, { exclusiveStartKey: 'key0', prefix: 'img/' });- expect(mockListKeys).toHaveBeenNthCalledWith(2, { exclusiveStartKey: 'key2', prefix: 'img/' });- expect(mockListKeys).toHaveBeenNthCalledWith(3, { exclusiveStartKey: 'key4', prefix: 'img/' });+ expect(mockListKeys).toHaveBeenCalledTimes(1);expect(results).toHaveLength(5);results.forEach((r, i) => {@@ -681,5 +587,63 @@ describe('KeyValueStore', () => {['key2', { value: 2 }],]);});++ test('await keys() should return all keys as a flat array', async () => {+ const store = await KeyValueStore.open();++ const testData = {+ key1: { value: 1 },+ key2: { value: 2 },+ key3: { value: 3 },+ };++ for (const [key, value] of Object.entries(testData)) {+ await store.setValue(key, value);+ }++ const keys = await store.keys();++ expect(keys).toEqual(['key1', 'key2', 'key3']);+ });++ test('await values() should return all values as a flat array', async () => {+ const store = await KeyValueStore.open();++ const testData = {+ key1: { value: 1 },+ key2: { value: 2 },+ key3: { value: 3 },+ };++ for (const [key, value] of Object.entries(testData)) {+ await store.setValue(key, value);+ }++ const values = await store.values<{ value: number }>();++ expect(values).toEqual([{ value: 1 }, { value: 2 }, { value: 3 }]);+ });++ test('await entries() should return all entries as a flat array', async () => {+ const store = await KeyValueStore.open();++ const testData = {+ key1: { value: 1 },+ key2: { value: 2 },+ key3: { value: 3 },+ };++ for (const [key, value] of Object.entries(testData)) {+ await store.setValue(key, value);+ }++ const entries = await store.entries<{ value: number }>();++ expect(entries).toEqual([+ ['key1', { value: 1 }],+ ['key2', { value: 2 }],+ ['key3', { value: 3 }],+ ]);+ });});});diff --git a/test/core/storages/request_queue.test.ts b/test/core/storages/request_queue.test.tsindex bb24fb142..2402a8159 100644--- a/test/core/storages/request_queue.test.ts+++ b/test/core/storages/request_queue.test.ts@@ -2,7 +2,6 @@import {API_PROCESSED_REQUESTS_DELAY_MILLIS,- Configuration,ProxyConfiguration,QUERY_HEAD_MIN_LENGTH,Request,@@ -12,8 +11,6 @@ import {STORAGE_CONSISTENCY_DELAY_MILLIS,} from '@crawlee/core';import { sleep } from '@crawlee/utils';-import { gotScraping } from 'got-scraping';-import type { MockedFunction } from 'vitest';import { MemoryStorageEmulator } from '../../shared/MemoryStorageEmulator.js';@@ -64,8 +61,8 @@ describe('RequestQueue remote', () => {});expect(queue['queueHeadIds'].length()).toBe(1);- expect(mockAddRequest).toBeCalledTimes(1);- expect(mockAddRequest).toBeCalledWith(requestA, { forefront: false });+ expect(mockAddRequest).toHaveBeenCalledTimes(1);+ expect(mockAddRequest).toHaveBeenCalledWith(requestA, { forefront: false });// Try to add again a request with the same URLconst queueOperationInfo2 = await queue.addRequest(requestOptions);@@ -86,7 +83,7 @@ describe('RequestQueue remote', () => {mockAddRequest.mockResolvedValueOnce(secondResolveValue);await queue.addRequest(requestB, { forefront: true });- expect(mockAddRequest).toBeCalledTimes(2);+ expect(mockAddRequest).toHaveBeenCalledTimes(2);expect(mockAddRequest).toHaveBeenLastCalledWith(requestB, { forefront: true });expect(queue['queueHeadIds'].length()).toBe(2);@@ -97,7 +94,7 @@ describe('RequestQueue remote', () => {mockGetRequest.mockResolvedValueOnce({ ...requestB, id: 'b' });const requestBFromQueue = await queue.fetchNextRequest();- expect(mockGetRequest).toBeCalledTimes(1);+ expect(mockGetRequest).toHaveBeenCalledTimes(1);expect(mockGetRequest).toHaveBeenLastCalledWith('b');expect(requestBFromQueue).toEqual({ ...requestB, id: 'b' });@@ -117,7 +114,7 @@ describe('RequestQueue remote', () => {mockGetRequest.mockResolvedValueOnce(undefined);const requestXFromQueue = await queue.getRequest('non-existent');- expect(mockGetRequest).toBeCalledTimes(2);+ expect(mockGetRequest).toHaveBeenCalledTimes(2);expect(mockGetRequest).toHaveBeenLastCalledWith('non-existent');expect(requestXFromQueue).toBe(null);@@ -133,7 +130,7 @@ describe('RequestQueue remote', () => {});await queue.reclaimRequest(requestBFromQueue!, { forefront: true });- expect(mockUpdateRequest).toBeCalledTimes(1);+ expect(mockUpdateRequest).toHaveBeenCalledTimes(1);expect(mockUpdateRequest).toHaveBeenLastCalledWith(requestBFromQueue, { forefront: true });expect(queue['queueHeadIds'].length()).toBe(1);@@ -147,7 +144,7 @@ describe('RequestQueue remote', () => {mockGetRequest.mockResolvedValueOnce(requestBFromQueue as never);const requestBFromQueue2 = await queue.fetchNextRequest();- expect(mockGetRequest).toBeCalledTimes(3);+ expect(mockGetRequest).toHaveBeenCalledTimes(3);expect(mockGetRequest).toHaveBeenLastCalledWith('b');expect(requestBFromQueue2).toEqual(requestBFromQueue);@@ -165,7 +162,7 @@ describe('RequestQueue remote', () => {});await queue.markRequestHandled(requestBFromQueue!);- expect(mockUpdateRequest).toBeCalledTimes(2);+ expect(mockUpdateRequest).toHaveBeenCalledTimes(2);expect(mockUpdateRequest).toHaveBeenLastCalledWith(requestBFromQueue);expect(queue['queueHeadIds'].length()).toBe(1);@@ -186,9 +183,9 @@ describe('RequestQueue remote', () => {mockGetRequest.mockResolvedValueOnce({ ...requestA, id: 'a' });const requestAFromQueue = await queue.fetchNextRequest();- expect(mockGetRequest).toBeCalledTimes(4);+ expect(mockGetRequest).toHaveBeenCalledTimes(4);expect(mockGetRequest).toHaveBeenLastCalledWith('a');- expect(mockListHead).toBeCalledTimes(1);+ expect(mockListHead).toHaveBeenCalledTimes(1);expect(mockListHead).toHaveBeenLastCalledWith({ limit: QUERY_HEAD_MIN_LENGTH });expect(requestAFromQueue).toEqual({ ...requestA, id: 'a' });@@ -196,12 +193,12 @@ describe('RequestQueue remote', () => {expect(queue.inProgressCount()).toBe(1);// Drop queue.- const mockDelete = vitest.spyOn(queue.client, 'delete');- mockDelete.mockResolvedValueOnce(undefined);+ const mockDrop = vitest.spyOn(queue.client, 'drop');+ mockDrop.mockResolvedValueOnce(undefined);await queue.drop();- expect(mockDelete).toBeCalledTimes(1);- expect(mockDelete).toHaveBeenLastCalledWith();+ expect(mockDrop).toHaveBeenCalledTimes(1);+ expect(mockDrop).toHaveBeenLastCalledWith();});test('addRequests', async () => {@@ -233,8 +230,8 @@ describe('RequestQueue remote', () => {// Ensure the client method was actually called, and addedexpect(queue['queueHeadIds'].length()).toBe(1);- expect(mockAddRequests).toBeCalledTimes(1);- expect(mockAddRequests).toBeCalledWith([requestA], { forefront: false });+ expect(mockAddRequests).toHaveBeenCalledTimes(1);+ expect(mockAddRequests).toHaveBeenCalledWith([requestA], { forefront: false });// Try to add a request with the same URL again, expecting cachedconst addRequestsResult2 = await queue.addRequests([requestOptions]);@@ -285,7 +282,7 @@ describe('RequestQueue remote', () => {expect(queue['queueHeadIds'].length()).toBe(3);expect(mockAddRequests).toHaveBeenCalled();- expect(mockAddRequests).toBeCalledWith([requestB, requestC], { forefront: true });+ expect(mockAddRequests).toHaveBeenCalledWith([requestB, requestC], { forefront: true });});test('addRequestsBatched does not retry permanently unprocessed requests forever', async () => {@@ -368,13 +365,13 @@ describe('RequestQueue remote', () => {});await queue.addRequest(requestA);- expect(addRequestMock).toBeCalledTimes(1);+ expect(addRequestMock).toHaveBeenCalledTimes(1);expect(addRequestMock).toHaveBeenLastCalledWith(requestA, { forefront: false });// Add request B that has same unique so that addRequest() is not called because it's already cached.// mock.expects('addRequest').never();const queueOperationInfo = await queue.addRequest(requestB);- expect(addRequestMock).toBeCalledTimes(1);+ expect(addRequestMock).toHaveBeenCalledTimes(1);expect(queueOperationInfo).toEqual({requestId: 'a',uniqueKey: requestA.uniqueKey,@@ -399,13 +396,13 @@ describe('RequestQueue remote', () => {});await queue.addRequest(requestX);- expect(addRequestMock).toBeCalledTimes(1);+ expect(addRequestMock).toHaveBeenCalledTimes(1);expect(addRequestMock).toHaveBeenLastCalledWith(requestX, { forefront: false });// Add request Y that has same unique so that addRequest() is not called because it's already cached.// mock.expects('addRequest').never();const queueOperationInfo = await queue.addRequest(requestY);- expect(addRequestMock).toBeCalledTimes(1);+ expect(addRequestMock).toHaveBeenCalledTimes(1);expect(queueOperationInfo).toEqual({requestId: 'x',uniqueKey: requestX.uniqueKey,@@ -425,7 +422,7 @@ describe('RequestQueue remote', () => {} as never);expect(await queue.isEmpty()).toBe(false);- expect(listHeadMock).toBeCalledTimes(1);+ expect(listHeadMock).toHaveBeenCalledTimes(1);expect(listHeadMock).toHaveBeenLastCalledWith({ limit: QUERY_HEAD_MIN_LENGTH });// Add request A and addRequest is not called because was already cached.@@ -433,7 +430,7 @@ describe('RequestQueue remote', () => {const addRequestMock = vitest.spyOn(queue.client, 'addRequest');const queueOperationInfo = await queue.addRequest(requestA);- expect(addRequestMock).toBeCalledTimes(0);+ expect(addRequestMock).toHaveBeenCalledTimes(0);expect(queueOperationInfo).toEqual({requestId: 'a',uniqueKey: 'aaa',@@ -458,7 +455,7 @@ describe('RequestQueue remote', () => {});await queue.addRequest(requestA, { forefront: true });- expect(addRequestMock).toBeCalledTimes(1);+ expect(addRequestMock).toHaveBeenCalledTimes(1);expect(addRequestMock).toHaveBeenLastCalledWith(requestA, { forefront: true });expect(queue['queueHeadIds'].length()).toBe(1);@@ -468,13 +465,13 @@ describe('RequestQueue remote', () => {getRequestMock.mockResolvedValueOnce(undefined);const fetchedRequest = await queue.fetchNextRequest();- expect(getRequestMock).toBeCalledTimes(1);+ expect(getRequestMock).toHaveBeenCalledTimes(1);expect(getRequestMock).toHaveBeenLastCalledWith('a');expect(fetchedRequest).toBe(null);// Give queue time to mark request 'a' as not in progressawait sleep(STORAGE_CONSISTENCY_DELAY_MILLIS + 10);- expect(listHeadMock).not.toBeCalled();+ expect(listHeadMock).not.toHaveBeenCalled();// Should try it once again (the queue head is queried again)getRequestMock.mockResolvedValueOnce({@@ -487,9 +484,9 @@ describe('RequestQueue remote', () => {} as never);const fetchedRequest2 = await queue.fetchNextRequest();- expect(getRequestMock).toBeCalledTimes(2);+ expect(getRequestMock).toHaveBeenCalledTimes(2);expect(getRequestMock).toHaveBeenLastCalledWith('a');- expect(listHeadMock).toBeCalledTimes(1);+ expect(listHeadMock).toHaveBeenCalledTimes(1);expect(listHeadMock).toHaveBeenLastCalledWith({ limit: QUERY_HEAD_MIN_LENGTH });expect(fetchedRequest2).toEqual({ ...requestA, id: 'a' });});@@ -514,12 +511,12 @@ describe('RequestQueue remote', () => {} as never);await queue.addRequest(requestA, { forefront: true });- expect(addRequestMock).toBeCalledTimes(1);+ expect(addRequestMock).toHaveBeenCalledTimes(1);expect(addRequestMock).toHaveBeenLastCalledWith(requestA, { forefront: true });const fetchedRequest = await queue.fetchNextRequest();- expect(getRequestMock).not.toBeCalled();- expect(listHeadMock).toBeCalledTimes(1);+ expect(getRequestMock).not.toHaveBeenCalled();+ expect(listHeadMock).toHaveBeenCalledTimes(1);expect(listHeadMock).toHaveBeenLastCalledWith({ limit: QUERY_HEAD_MIN_LENGTH });expect(fetchedRequest).toBe(null);});@@ -535,7 +532,7 @@ describe('RequestQueue remote', () => {const requestOpts = { url: 'http://example.com/a' };await queue.addRequest(requestOpts);- expect(addRequestMock).toBeCalledTimes(1);+ expect(addRequestMock).toHaveBeenCalledTimes(1);expect(addRequestMock).toHaveBeenLastCalledWith(new Request(requestOpts), { forefront: false });});@@ -547,7 +544,7 @@ describe('RequestQueue remote', () => {} as never);const count = await queue.handledCount();expect(count).toBe(33);- expect(getMock).toBeCalledTimes(1);+ expect(getMock).toHaveBeenCalledTimes(1);expect(getMock).toHaveBeenLastCalledWith();});@@ -571,7 +568,7 @@ describe('RequestQueue remote', () => {const isFinished = await queue.isFinished();expect(isFinished).toBe(true);- expect(listHeadMock).toBeCalledTimes(2);+ expect(listHeadMock).toHaveBeenCalledTimes(2);expect(listHeadMock).toHaveBeenNthCalledWith(1, { limit: QUERY_HEAD_MIN_LENGTH });expect(listHeadMock).toHaveBeenNthCalledWith(2, { limit: QUERY_HEAD_MIN_LENGTH });});@@ -599,7 +596,7 @@ describe('RequestQueue remote', () => {expect(queue.inProgressCount()).toBe(0);expect(queue.assumedTotalCount).toBe(2);expect(queue.assumedHandledCount).toBe(0);- expect(addRequestMock).toBeCalledTimes(2);+ expect(addRequestMock).toHaveBeenCalledTimes(2);expect(addRequestMock).toHaveBeenNthCalledWith(1, requestA, { forefront: true });expect(addRequestMock).toHaveBeenNthCalledWith(2, requestB, { forefront: true });@@ -608,7 +605,7 @@ describe('RequestQueue remote', () => {const isFinished = await queue.isFinished();expect(isFinished).toBe(false);- expect(listHeadMock).not.toBeCalled();+ expect(listHeadMock).not.toHaveBeenCalled();// Fetch them from queue.const getRequestMock = vitest.spyOn(queue.client, 'getRequest');@@ -617,11 +614,11 @@ describe('RequestQueue remote', () => {const requestBFromQueue = await queue.fetchNextRequest();expect(requestBFromQueue).toEqual(requestBWithId);- expect(getRequestMock).toBeCalledTimes(1);+ expect(getRequestMock).toHaveBeenCalledTimes(1);expect(getRequestMock).toHaveBeenLastCalledWith('b');const requestAFromQueue = await queue.fetchNextRequest();expect(requestAFromQueue).toEqual(requestAWithId);- expect(getRequestMock).toBeCalledTimes(2);+ expect(getRequestMock).toHaveBeenCalledTimes(2);expect(getRequestMock).toHaveBeenLastCalledWith('a');expect(queue['queueHeadIds'].length()).toBe(0);@@ -631,20 +628,20 @@ describe('RequestQueue remote', () => {// It won't query the head as there is something in progress or pending.expect(await queue.isFinished()).toBe(false);- expect(listHeadMock).not.toBeCalled();+ expect(listHeadMock).not.toHaveBeenCalled();// Reclaim one and mark another one handled.const updateRequestMock = vitest.spyOn(queue.client, 'updateRequest');updateRequestMock.mockResolvedValueOnce({ requestId: 'b', wasAlreadyHandled: false, wasAlreadyPresent: true });await queue.markRequestHandled(requestBWithId);- expect(updateRequestMock).toBeCalledTimes(1);+ expect(updateRequestMock).toHaveBeenCalledTimes(1);expect(updateRequestMock).toHaveBeenLastCalledWith(requestBWithId);updateRequestMock.mockResolvedValueOnce({ requestId: 'a', wasAlreadyHandled: false, wasAlreadyPresent: true });await queue.reclaimRequest(requestAWithId, { forefront: true });- expect(updateRequestMock).toBeCalledTimes(2);+ expect(updateRequestMock).toHaveBeenCalledTimes(2);expect(updateRequestMock).toHaveBeenLastCalledWith(requestAWithId, { forefront: true });expect(queue['queueHeadIds'].length()).toBe(0);@@ -660,7 +657,7 @@ describe('RequestQueue remote', () => {// It won't query the head as there is something in progress or pending.expect(await queue.isFinished()).toBe(false);- expect(listHeadMock).not.toBeCalled();+ expect(listHeadMock).not.toHaveBeenCalled();// Fetch again.// @ts-expect-error Argument of type 'Request' is not assignable to parameter of type@@ -669,7 +666,7 @@ describe('RequestQueue remote', () => {const requestAFromQueue2 = await queue.fetchNextRequest();expect(requestAFromQueue2).toEqual(requestAWithId);- expect(getRequestMock).toBeCalledTimes(3);+ expect(getRequestMock).toHaveBeenCalledTimes(3);expect(getRequestMock).toHaveBeenLastCalledWith('a');expect(queue['queueHeadIds'].length()).toBe(0);@@ -679,13 +676,13 @@ describe('RequestQueue remote', () => {// It won't query the head as there is something in progress or pending.expect(await queue.isFinished()).toBe(false);- expect(listHeadMock).not.toBeCalled();+ expect(listHeadMock).not.toHaveBeenCalled();// Mark handled.updateRequestMock.mockResolvedValueOnce({ requestId: 'a', wasAlreadyHandled: false, wasAlreadyPresent: true });await queue.markRequestHandled(requestAWithId);- expect(updateRequestMock).toBeCalledTimes(3);+ expect(updateRequestMock).toHaveBeenCalledTimes(3);expect(updateRequestMock).toHaveBeenLastCalledWith(requestAWithId);expect(queue['queueHeadIds'].length()).toBe(0);@@ -703,7 +700,7 @@ describe('RequestQueue remote', () => {});expect(await queue.isFinished()).toBe(true);- expect(listHeadMock).toBeCalledTimes(1);+ expect(listHeadMock).toHaveBeenCalledTimes(1);expect(listHeadMock).toHaveBeenLastCalledWith({ limit: QUERY_HEAD_MIN_LENGTH });});@@ -764,17 +761,17 @@ describe('RequestQueue remote', () => {const result = await queue.getInfo();expect(result).toEqual(expected);- expect(getMock).toBeCalledTimes(1);+ expect(getMock).toHaveBeenCalledTimes(1);expect(getMock).toHaveBeenLastCalledWith();});test('drop() works', async () => {const queue = await createRequestQueue('some-id', 'some-name');- const deleteMock = vitest.spyOn(queue.client, 'delete').mockResolvedValueOnce(undefined);+ const dropMock = vitest.spyOn(queue.client, 'drop').mockResolvedValueOnce(undefined);await queue.drop();- expect(deleteMock).toBeCalledTimes(1);- expect(deleteMock).toHaveBeenLastCalledWith();+ expect(dropMock).toHaveBeenCalledTimes(1);+ expect(dropMock).toHaveBeenLastCalledWith();});test('Request.userData.__crawlee internal object is non-enumerable and always defined', async () => {@@ -851,9 +848,9 @@ describe('RequestQueue with requestsFromUrl', () => {expect(await queue.fetchNextRequest()).toMatchObject({ method: 'POST', url: list2[0] });expect(await queue.fetchNextRequest()).toMatchObject({ method: 'POST', url: list2[1] });- expect(spy).toBeCalledTimes(2);- expect(spy).toBeCalledWith({ url: 'http://example.com/list-1', urlRegExp: undefined });- expect(spy).toBeCalledWith({ url: 'http://example.com/list-2', urlRegExp: undefined });+ expect(spy).toHaveBeenCalledTimes(2);+ expect(spy).toHaveBeenCalledWith({ url: 'http://example.com/list-1', urlRegExp: undefined });+ expect(spy).toHaveBeenCalledWith({ url: 'http://example.com/list-2', urlRegExp: undefined });});test('should use regex parameter to parse urls', async () => {@@ -876,7 +873,7 @@ describe('RequestQueue with requestsFromUrl', () => {expect(await queue.fetchNextRequest()).toMatchObject({ method: 'GET', url: listArr[1] });await queue.drop();- expect(mockHttpClient.sendRequest).toBeCalled();+ expect(mockHttpClient.sendRequest).toHaveBeenCalled();expect(mockHttpClient.sendRequest.mock.calls[0][0].url).toBe('http://example.com/list-1');});@@ -920,8 +917,8 @@ describe('RequestQueue with requestsFromUrl', () => {expect(await queue.fetchNextRequest()).toBe(null);- expect(spy).toBeCalledTimes(1);- expect(spy).toBeCalledWith({ url: 'http://example.com/list-1', urlRegExp: undefined });+ expect(spy).toHaveBeenCalledTimes(1);+ expect(spy).toHaveBeenCalledWith({ url: 'http://example.com/list-1', urlRegExp: undefined });});test('should use the defined proxy server when using `requestsFromUrl`', async () => {@@ -941,7 +938,7 @@ describe('RequestQueue with requestsFromUrl', () => {{ requestsFromUrl: 'http://example.com/list-3' },]);- expect(spy).not.toBeCalledWith(expect.not.objectContaining({ proxyUrl: expect.any(String) }));+ expect(spy).not.toHaveBeenCalledWith(expect.not.objectContaining({ proxyUrl: expect.any(String) }));});});diff --git a/test/shared/MemoryStorageEmulator.ts b/test/shared/MemoryStorageEmulator.tsindex ba1537b64..c48a826c9 100644--- a/test/shared/MemoryStorageEmulator.ts+++ b/test/shared/MemoryStorageEmulator.ts@@ -36,7 +36,7 @@ export class MemoryStorageEmulator extends StorageEmulator {async getDatasetItems(id?: string) {const dataset = await this.getDataset(id);- return (await dataset.listItems()).items;+ return (await dataset.getData()).items;}getRequestQueue(id?: string) {@@ -54,7 +54,7 @@ export class MemoryStorageEmulator extends StorageEmulator {}async getState() {- return await (await this.getKeyValueStore()).getRecord('CRAWLEE_STATE');+ return await (await this.getKeyValueStore()).getValue('CRAWLEE_STATE');}}