Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

refactor!: Rename `StorageClient` to `StorageBackend` (#3825)

Jan Buchard7e44dcc5237
diff --git a/REBASE-DEFERRED.md b/REBASE-DEFERRED.md
index 8dacdc617..9cf562e13 100644
--- a/REBASE-DEFERRED.md
+++ b/REBASE-DEFERRED.md
@@ -484,3 +484,34 @@ awaits one microtask via `Promise.resolve(undefined)` racing `sleep(1)`), but
worth a final look at whether it should be simplified now that the
underlying async-flush problem it solved no longer applies in this
architecture.
+
+**Commit `eec66ca55` (Rename StorageClient to StorageBackend)**: besides the
+3 flagged conflicts (`snapshotter.ts`, `basic_crawler.test.ts`,
+`playwright_crawler.test.ts`), found several unconflicted-but-broken spots
+this rename missed because they were added by v4-only or my-own-authored code
+after this commit's original point in v4 history:
+- `packages/core/src/autoscaling/client_load_signal.ts` still imported/typed
+ against `StorageClient` (removed from `@crawlee/types` entirely) — fixed to
+ `StorageBackend`.
+- `packages/core/src/autoscaling/snapshotter.ts` had 3 more unconflicted
+ `StorageClient`/`getStorageClient` references outside the merge hunk — fixed.
+- `test/core/storages/storage_manager.test.ts` was untouched by any conflict
+ in this entire 130-commit rebase and had gone fully stale: still imported
+ the deleted `MemoryStorageEmulator` helper (removed 2 commits ago in
+ `a2f9b2646`) and called `Configuration.getStorageClient()`, a method that
+ doesn't exist on `Configuration` at all post the earlier ServiceLocator
+ redesign. Rewrote against the current API
+ (`serviceLocator.setStorageBackend(new MemoryStorageBackend())`,
+ `Dataset.open(id, { storageBackend })`).
+- The two path-traversal regression tests I wrote earlier this session
+ (`packages/fs-storage/test/storage-name-path-traversal.test.ts`,
+ `.../key-value-store/record-key-path-traversal.test.ts`) used the
+ pre-rename `FileSystemStorageClient`/`createXClient` names — updated to
+ `FileSystemStorageBackend`/`createXBackend`. Also removed the now-gone
+ `writeMetadata` constructor option and, in the record-key test, stopped
+ asserting raw on-disk file paths — the previous commit's rewrite
+ (`8a422628f`) delegates file naming/encoding to the native
+ `@crawlee/fs-storage-native` client, and its own `special-keys.test.ts`
+ explicitly treats on-disk names as an implementation detail not to assert
+ on. Both tests still assert the actual security property (escaping key/name
+ rejected) through the public API only.
diff --git a/docs/guides/configuration.mdx b/docs/guides/configuration.mdx
index 8ec11c466..b93727c44 100644
--- a/docs/guides/configuration.mdx
+++ b/docs/guides/configuration.mdx
@@ -137,7 +137,7 @@ The last option to adjust Crawlee configuration is to use the <ApiLink to="core/
### Global Configuration
-By default, there is a global singleton instance of `Configuration` class, it is used by the crawlers and some other classes that depend on a configurable behavior. In most cases you don't need to adjust any options there, but if needed - you can access it via <ApiLink to="core/class/Configuration#getGlobalConfig">`Configuration.getGlobalConfig()`</ApiLink>, which delegates to the global <ApiLink to="core/class/ServiceLocator">`serviceLocator`</ApiLink> — the single source of truth for Crawlee's shared services (for example the configuration, event manager, storage client, and logger). You can also reach the same instance directly via `serviceLocator.getConfiguration()` or swap services globally with `serviceLocator.setConfiguration(...)` before any crawler is created. Configuration values are accessible directly as properties on the instance.
+By default, there is a global singleton instance of `Configuration` class, it is used by the crawlers and some other classes that depend on a configurable behavior. In most cases you don't need to adjust any options there, but if needed - you can access it via <ApiLink to="core/class/Configuration#getGlobalConfig">`Configuration.getGlobalConfig()`</ApiLink>, which delegates to the global <ApiLink to="core/class/ServiceLocator">`serviceLocator`</ApiLink> — the single source of truth for Crawlee's shared services (for example the configuration, event manager, storage backend, and logger). You can also reach the same instance directly via `serviceLocator.getConfiguration()` or swap services globally with `serviceLocator.setConfiguration(...)` before any crawler is created. Configuration values are accessible directly as properties on the instance.
```js
import { CheerioCrawler, Configuration, sleep } from 'crawlee';
diff --git a/docs/guides/parallel-scraping/parallel-scraper.mjs b/docs/guides/parallel-scraping/parallel-scraper.mjs
index 08683a4a4..3b2fe80a1 100644
--- a/docs/guides/parallel-scraping/parallel-scraper.mjs
+++ b/docs/guides/parallel-scraping/parallel-scraper.mjs
@@ -1,6 +1,6 @@
import { fork } from 'node:child_process';
-import { FileSystemStorageClient } from '@crawlee/fs-storage';
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
import { Configuration, Dataset, PlaywrightCrawler, log } from 'crawlee';
import { router } from './routes.mjs';
@@ -85,8 +85,8 @@ if (!process.env.IN_WORKER_THREAD) {
// Store the worker's own internal state (its default dataset, key-value store, etc.) in a separate
// directory so the workers don't collide with each other. This directory is private to a single
// worker, so we set `requestQueueAccess: 'single'` — the concurrency-safe locking only matters for
- // the shared `shop-urls` queue, which gets its own storage client in `requestQueue.mjs`.
- const storageClient = new FileSystemStorageClient({
+ // the shared `shop-urls` queue, which gets its own storage backend in `requestQueue.mjs`.
+ const storageBackend = new FileSystemStorageBackend({
localDataDirectory: `./storage/worker-${process.env.WORKER_INDEX}`,
requestQueueAccess: 'single',
});
@@ -104,9 +104,9 @@ if (!process.env.IN_WORKER_THREAD) {
// highlight-end
// Let's also limit the crawler's concurrency, we don't want to overload a single process 🐌
maxConcurrency: 5,
- // Use the worker-specific, concurrency-safe storage client we created above
+ // Use the worker-specific, concurrency-safe storage backend we created above
// highlight-start
- storageClient,
+ storageBackend,
// highlight-end
},
config,
diff --git a/docs/guides/parallel-scraping/parallel-scraping.mdx b/docs/guides/parallel-scraping/parallel-scraping.mdx
index ca7cc571e..82c0b3be9 100644
--- a/docs/guides/parallel-scraping/parallel-scraping.mdx
+++ b/docs/guides/parallel-scraping/parallel-scraping.mdx
@@ -62,9 +62,9 @@ The exported function, `getOrInitQueue`, might seem like it does a lot. In essen
:::caution Make the shared queue concurrency-safe with `requestQueueAccess: 'shared'`
-Because every worker process opens this same `shop-urls` queue at the same time, it **must** use the concurrency-safe locking behavior of `FileSystemStorageClient`. That's why `getOrInitQueue` opens the queue with a storage client constructed with `requestQueueAccess: 'shared'`.
+Because every worker process opens this same `shop-urls` queue at the same time, it **must** use the concurrency-safe locking behavior of `FileSystemStorageBackend`. That's why `getOrInitQueue` opens the queue with a storage backend constructed with `requestQueueAccess: 'shared'`.
-By default, `FileSystemStorageClient` assumes it is the *sole* consumer of a queue (`requestQueueAccess: 'single'`). On open it immediately reclaims any requests left *in progress* — great for a single-process crawl recovering after a crash, but disastrous when workers run side by side: each worker would happily grab requests another worker is still processing, so the same URL gets scraped multiple times.
+By default, `FileSystemStorageBackend` assumes it is the *sole* consumer of a queue (`requestQueueAccess: 'single'`). On open it immediately reclaims any requests left *in progress* — great for a single-process crawl recovering after a crash, but disastrous when workers run side by side: each worker would happily grab requests another worker is still processing, so the same URL gets scraped multiple times.
Setting `requestQueueAccess: 'shared'` tells the client to treat an in-progress request as a potential live peer's lock and only reclaim it once the lock expires on the wall clock, so two workers never process the same request at once.
@@ -132,17 +132,17 @@ This will check how the script is executed as. If this value has _any_ value, it
We use this to ensure the parent process stays alive until all the worker processes exit. Otherwise, the worker processes would just get spawned, and lose the ability to communicate with the parent. You might not need this depending on your use case (maybe you just need to spawn workers and let them process).
-#### What's with all the `Configuration` and storage client setup?
+#### What's with all the `Configuration` and storage backend setup?
There are two things we want to do for the worker processes:
-- get the shared queue from the same location as the parent process (it already comes with the concurrency-safe storage client we set up in `requestQueue.mjs`)
+- get the shared queue from the same location as the parent process (it already comes with the concurrency-safe storage backend we set up in `requestQueue.mjs`)
- ensure the default storages do **not** get purged on start, as otherwise we'd lose the queue we prepared, and give each worker its own private storage directory for its internal state so the workers don't collide with each other
In order, that's what these lines do:
```javascript title="src/parallel-scraper.mjs"
-import { FileSystemStorageClient } from '@crawlee/fs-storage';
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
// Get the shared request queue from the parent process (step 1)
const requestQueue = await getOrInitQueue(false);
@@ -153,7 +153,7 @@ const config = new Configuration({ purgeOnStart: false });
// Store the worker's own internal state in a separate directory so workers don't collide (step 2,
// cont.). This directory is private to a single worker, so we explicitly set
// `requestQueueAccess: 'single'`.
-const storageClient = new FileSystemStorageClient({
+const storageBackend = new FileSystemStorageBackend({
localDataDirectory: `./storage/worker-${process.env.WORKER_INDEX}`,
requestQueueAccess: 'single',
});
@@ -169,7 +169,7 @@ Each worker's `./storage/worker-N` directory is private to that single worker
You might have noticed several lines highlighted in the code above. Those show how you provide the shared request queue to the crawler.
-You might have also noticed we passed in the `config` and `storageClient` we created earlier to the crawler. These ensure the crawler uses the worker-specific storages for its own internal state (so the workers do not collide with each other), while still consuming the shared, concurrency-safe `shop-urls` queue we provided explicitly.
+You might have also noticed we passed in the `config` and `storageBackend` we created earlier to the crawler. These ensure the crawler uses the worker-specific storages for its own internal state (so the workers do not collide with each other), while still consuming the shared, concurrency-safe `shop-urls` queue we provided explicitly.
#### Why do we use `process.send` instead of `context.pushData`?
diff --git a/docs/guides/parallel-scraping/shared.mjs b/docs/guides/parallel-scraping/shared.mjs
index 573f97481..bef086bfb 100644
--- a/docs/guides/parallel-scraping/shared.mjs
+++ b/docs/guides/parallel-scraping/shared.mjs
@@ -1,4 +1,4 @@
-import { FileSystemStorageClient } from '@crawlee/fs-storage';
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
import { RequestQueue } from 'crawlee';
// The request queue shared by all the parallel workers
@@ -9,7 +9,7 @@ let queue;
// is still processing is treated as a live peer's lock and is not handed out again until that lock
// expires — so two workers never scrape the same URL at once. (We point at the `./storage`
// location, which is where this shared queue lives.)
-const sharedStorageClient = new FileSystemStorageClient({
+const sharedStorageBackend = new FileSystemStorageBackend({
localDataDirectory: './storage',
requestQueueAccess: 'shared',
});
@@ -23,11 +23,11 @@ export async function getOrInitQueue(makeFresh = false) {
return queue;
}
- queue = await RequestQueue.open('shop-urls', { storageClient: sharedStorageClient });
+ queue = await RequestQueue.open('shop-urls', { storageBackend: sharedStorageBackend });
if (makeFresh) {
await queue.drop();
- queue = await RequestQueue.open('shop-urls', { storageClient: sharedStorageClient });
+ queue = await RequestQueue.open('shop-urls', { storageBackend: sharedStorageBackend });
}
return queue;
diff --git a/docs/guides/request_storage.mdx b/docs/guides/request_storage.mdx
index 49ea41359..42c49ef37 100644
--- a/docs/guides/request_storage.mdx
+++ b/docs/guides/request_storage.mdx
@@ -26,7 +26,7 @@ Each Crawlee project run is associated with a **default request queue**. Typical
In Crawlee, the request queue is represented by the <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink> class.
-By default, the request queue is managed by the <ApiLink to="fs-storage/class/FileSystemStorageClient">`FileSystemStorageClient`</ApiLink> class and its data is stored in the local directory specified by the `CRAWLEE_STORAGE_DIR` environment variable as follows:
+By default, the request queue is managed by the <ApiLink to="fs-storage/class/FileSystemStorageBackend">`FileSystemStorageBackend`</ApiLink> class and its data is stored in the local directory specified by the `CRAWLEE_STORAGE_DIR` environment variable as follows:
```text
{CRAWLEE_STORAGE_DIR}/request_queues/{QUEUE_ID}/entries.json
@@ -88,4 +88,4 @@ import { purgeDefaultStorages } from 'crawlee';
await purgeDefaultStorages();
```
-Calling this function will clean up the default request storage directory (and also the request list stored in default key-value store). This is a shortcut for running (optional) `purge` method on the <ApiLink to="core/interface/StorageClient">`StorageClient`</ApiLink> interface, in other words it will call the `purge` method of the underlying storage implementation we are currently using. You can make sure the storage is purged only once for a given execution context if you set `onlyPurgeOnce` to `true` in the `options` object.
+Calling this function will clean up the default request storage directory (and also the request list stored in default key-value store). This is a shortcut for running (optional) `purge` method on the <ApiLink to="core/interface/StorageBackend">`StorageBackend`</ApiLink> interface, in other words it will call the `purge` method of the underlying storage implementation we are currently using. You can make sure the storage is purged only once for a given execution context if you set `onlyPurgeOnce` to `true` in the `options` object.
diff --git a/docs/guides/result_storage.mdx b/docs/guides/result_storage.mdx
index 583f613c6..eb2b4ac50 100644
--- a/docs/guides/result_storage.mdx
+++ b/docs/guides/result_storage.mdx
@@ -8,7 +8,7 @@ import ApiLink from '@site/src/components/ApiLink';
Crawlee has several result storage types that are useful for specific tasks. The data is stored on a local disk to the directory defined by the `CRAWLEE_STORAGE_DIR` environment variable. If this variable is not defined, by default Crawlee sets `CRAWLEE_STORAGE_DIR` to `./storage` in the current working directory.
-By default, Crawlee storage is managed by the <ApiLink to="fs-storage/class/FileSystemStorageClient">`FileSystemStorageClient`</ApiLink> class, which stores all information as local files in the respective storage type folders.
+By default, Crawlee storage is managed by the <ApiLink to="fs-storage/class/FileSystemStorageBackend">`FileSystemStorageBackend`</ApiLink> class, which stores all information as local files in the respective storage type folders.
## Key-value store
@@ -110,4 +110,4 @@ import { purgeDefaultStorages } from 'crawlee';
await purgeDefaultStorages();
```
-Calling this function will clean up the default results storage directories except the `INPUT` key in default key-value store directory. This is a shortcut for running (optional) `purge` method on the <ApiLink to="core/interface/StorageClient">`StorageClient`</ApiLink> interface, in other words it will call the `purge` method of the underlying storage implementation we are currently using. In addition, this method will make sure the storage is purged only once for a given execution context, so it is safe to call it multiple times.
+Calling this function will clean up the default results storage directories except the `INPUT` key in default key-value store directory. This is a shortcut for running (optional) `purge` method on the <ApiLink to="core/interface/StorageBackend">`StorageBackend`</ApiLink> interface, in other words it will call the `purge` method of the underlying storage implementation we are currently using. In addition, this method will make sure the storage is purged only once for a given execution context, so it is safe to call it multiple times.
diff --git a/docs/public-api/crawlee-basic.api.md b/docs/public-api/crawlee-basic.api.md
index 6b0680b3b..3adf0364d 100644
--- a/docs/public-api/crawlee-basic.api.md
+++ b/docs/public-api/crawlee-basic.api.md
@@ -53,7 +53,7 @@ import type { Source } from '@crawlee/core';
import { Statistics } from '@crawlee/core';
import type { StatisticsOptions } from '@crawlee/core';
import type { StatisticState } from '@crawlee/core';
-import type { StorageClient } from '@crawlee/types';
+import type { StorageBackend } from '@crawlee/types';
import type { StorageIdentifier } from '@crawlee/core';
import { StringPredicate } from 'ow';
import { TimeoutError } from '@apify/timeout';
@@ -167,7 +167,7 @@ export class BasicCrawler<Context extends CrawlingContext = CrawlingContext, Con
onSkippedRequest: Predicate<Function> & BasePredicate<Function | undefined>;
httpClient: ObjectPredicate<object> & BasePredicate<object | undefined>;
configuration: ObjectPredicate<object> & BasePredicate<object | undefined>;
- storageClient: ObjectPredicate<object> & BasePredicate<object | undefined>;
+ storageBackend: ObjectPredicate<object> & BasePredicate<object | undefined>;
eventManager: ObjectPredicate<object> & BasePredicate<object | undefined>;
logger: ObjectPredicate<object> & BasePredicate<object | undefined>;
minConcurrency: NumberPredicate & BasePredicate<number | undefined>;
@@ -260,7 +260,7 @@ export interface BasicCrawlerOptions<Context extends CrawlingContext = CrawlingC
statisticsOptions?: StatisticsOptions;
statusMessageCallback?: StatusMessageCallback;
statusMessageLoggingInterval?: number;
- storageClient?: StorageClient;
+ storageBackend?: StorageBackend;
}
// @public (undocumented)
diff --git a/docs/public-api/crawlee-core.api.md b/docs/public-api/crawlee-core.api.md
index 903290b39..5bf838f0a 100644
--- a/docs/public-api/crawlee-core.api.md
+++ b/docs/public-api/crawlee-core.api.md
@@ -17,13 +17,13 @@ import { Cookie as Cookie_2 } from 'tough-cookie';
import { CookieJar } from 'tough-cookie';
import { CrawleeLogger } from '@crawlee/types';
import type { CrawleeLoggerOptions } from '@crawlee/types';
-import type { DatasetClient } from '@crawlee/types';
+import type { DatasetBackend } from '@crawlee/types';
import type { DatasetInfo } from '@crawlee/types';
import { Dictionary } from '@crawlee/types';
import type { HttpRequestOptions } from '@crawlee/types';
import type { ISession } from '@crawlee/types';
import type { ISessionPool } from '@crawlee/types';
-import type { KeyValueStoreClient } from '@crawlee/types';
+import type { KeyValueStoreBackend } from '@crawlee/types';
import { Log } from '@apify/log';
import log from '@apify/log';
import { Logger } from '@apify/log';
@@ -39,7 +39,7 @@ import { PseudoUrl } from '@apify/pseudo_url';
import { QueueOperationInfo } from '@crawlee/types';
import { Readable } from 'node:stream';
import type { ReadonlyDeep } from 'type-fest';
-import type { RequestQueueClient } from '@crawlee/types';
+import type { RequestQueueBackend } from '@crawlee/types';
import type { RequestQueueInfo } from '@crawlee/types';
import { RobotsTxtFile } from '@crawlee/utils';
import type { SendRequestOptions } from '@crawlee/types';
@@ -47,7 +47,7 @@ import type { SessionFingerprint } from '@crawlee/types';
import { SessionState } from '@crawlee/types';
import type { SetRequired } from 'type-fest';
import type * as storage from '@crawlee/types';
-import { StorageClient } from '@crawlee/types';
+import { StorageBackend } from '@crawlee/types';
import { StorageIdentifier } from '@crawlee/types';
import { tryAbsoluteURL } from '@crawlee/utils';
import { z } from 'zod';
@@ -336,7 +336,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {
// @internal
constructor(options: DatasetOptions, config?: Configuration);
// (undocumented)
- client: DatasetClient<Data>;
+ backend: DatasetBackend<Data>;
// (undocumented)
readonly config: Configuration;
drop(): Promise<void>;
@@ -436,7 +436,7 @@ export interface DatasetMapper<Data, R> {
// @public (undocumented)
export interface DatasetOptions {
// (undocumented)
- client: DatasetClient;
+ backend: DatasetBackend;
// (undocumented)
id: string;
// (undocumented)
@@ -826,7 +826,7 @@ export interface KeyValueStoreIteratorOptions {
// @public (undocumented)
export interface KeyValueStoreOptions {
// (undocumented)
- client: KeyValueStoreClient;
+ backend: KeyValueStoreBackend;
// (undocumented)
id: string;
// (undocumented)
@@ -896,24 +896,24 @@ export const MAX_POOL_SIZE = 1000;
export const MAX_QUERIES_FOR_CONSISTENCY = 6;
// @public (undocumented)
-export class MemoryStorageClient implements storage.StorageClient {
+export class MemoryStorageBackend implements storage.StorageBackend {
constructor(options?: MemoryStorageOptions);
// (undocumented)
- createDatasetClient(options?: storage.CreateDatasetClientOptions): Promise<storage.DatasetClient>;
+ createDatasetBackend(options?: storage.CreateDatasetBackendOptions): Promise<storage.DatasetBackend>;
// (undocumented)
- createKeyValueStoreClient(options?: storage.CreateKeyValueStoreClientOptions): Promise<storage.KeyValueStoreClient>;
+ createKeyValueStoreBackend(options?: storage.CreateKeyValueStoreBackendOptions): Promise<storage.KeyValueStoreBackend>;
// (undocumented)
- createRequestQueueClient(options?: storage.CreateRequestQueueClientOptions): Promise<RequestQueueClient_2>;
+ createRequestQueueBackend(options?: storage.CreateRequestQueueBackendOptions): Promise<RequestQueueBackend_2>;
// (undocumented)
- readonly datasetClientCache: DatasetClient_2[];
- getStorageClientCacheKey(): string;
+ readonly datasetBackendCache: DatasetBackend_2[];
+ getStorageBackendCacheKey(): string;
// (undocumented)
- readonly keyValueStoreCache: KeyValueStoreClient_2[];
+ readonly keyValueStoreBackendCache: KeyValueStoreBackend_2[];
// (undocumented)
readonly logger?: CrawleeLogger;
purge(): Promise<void>;
// (undocumented)
- readonly requestQueueCache: RequestQueueClient_2[];
+ readonly requestQueueBackendCache: RequestQueueBackend_2[];
// (undocumented)
storageExists(id: string, type: 'Dataset' | 'KeyValueStore' | 'RequestQueue'): Promise<boolean>;
teardown(): Promise<void>;
@@ -1011,7 +1011,7 @@ export type PseudoUrlObject = {
export function purgeDefaultStorages(options?: PurgeDefaultStorageOptions): Promise<void>;
// @public
-export function purgeDefaultStorages(config?: Configuration, client?: StorageClient): Promise<void>;
+export function purgeDefaultStorages(config?: Configuration, storageBackend?: StorageBackend): Promise<void>;
// @public (undocumented)
export interface PushErrorMessageOptions {
@@ -1279,9 +1279,9 @@ export class RequestQueue implements IStorage, IRequestManager {
addRequest(requestLike: Source, options?: RequestQueueOperationOptions): Promise<RequestQueueOperationInfo>;
addRequests(requestsLike: RequestsLike, options?: RequestQueueOperationOptions): Promise<BatchAddRequestsResult>;
addRequestsBatched(requests: ReadonlyDeep<RequestsLike>, options?: AddRequestsBatchedOptions): Promise<AddRequestsBatchedResult>;
- protected _cacheRequest(cacheKey: string, queueOperationInfo: RequestQueueOperationInfo): void;
// (undocumented)
- client: RequestQueueClient;
+ backend: RequestQueueBackend;
+ protected _cacheRequest(cacheKey: string, queueOperationInfo: RequestQueueOperationInfo): void;
// (undocumented)
clientKey: string;
// (undocumented)
@@ -1343,7 +1343,7 @@ export interface RequestQueueOperationOptions {
// @public (undocumented)
export interface RequestQueueOptions {
// (undocumented)
- client: RequestQueueClient;
+ backend: RequestQueueBackend;
// (undocumented)
id: string;
// (undocumented)
@@ -1408,7 +1408,7 @@ export function resolveBaseUrlForEnqueueLinksFiltering(input: ResolveBaseUrl): s
export type ResolvedConfigValues = FieldsOutput<typeof crawleeConfigFields>;
// @public
-export function resolveStorageIdentifier(identifier: string | StorageIdentifier | null | undefined, client: StorageClient, storageType: 'Dataset' | 'KeyValueStore' | 'RequestQueue'): Promise<ExplicitStorageIdentifier>;
+export function resolveStorageIdentifier(identifier: string | StorageIdentifier | null | undefined, storageBackend: StorageBackend, storageType: 'Dataset' | 'KeyValueStore' | 'RequestQueue'): Promise<ExplicitStorageIdentifier>;
// @public (undocumented)
export interface ResponseLike {
@@ -1482,7 +1482,7 @@ export class ServiceConflictError extends Error {
// @public
export class ServiceLocator implements ServiceLocatorInterface {
- constructor(configuration?: Configuration, eventManager?: EventManager, storageClient?: StorageClient, logger?: CrawleeLogger);
+ constructor(configuration?: Configuration, eventManager?: EventManager, storageBackend?: StorageBackend, logger?: CrawleeLogger);
// (undocumented)
getChildLog(prefix: string): CrawleeLogger;
// (undocumented)
@@ -1492,7 +1492,7 @@ export class ServiceLocator implements ServiceLocatorInterface {
// (undocumented)
getLogger(): CrawleeLogger;
// (undocumented)
- getStorageClient(): StorageClient;
+ getStorageBackend(): StorageBackend;
// (undocumented)
getStorageInstanceManager(): StorageInstanceManager;
// (undocumented)
@@ -1504,7 +1504,7 @@ export class ServiceLocator implements ServiceLocatorInterface {
// (undocumented)
setLogger(logger: CrawleeLogger): void;
// (undocumented)
- setStorageClient(storageClient: StorageClient): void;
+ setStorageBackend(storageBackend: StorageBackend): void;
}
// @public (undocumented)
@@ -1896,7 +1896,7 @@ export interface StatisticState {
// @internal
export const STORAGE_CONSISTENCY_DELAY_MILLIS = 3000;
-export { StorageClient }
+export { StorageBackend }
export { StorageIdentifier }
@@ -1904,8 +1904,8 @@ export { StorageIdentifier }
export class StorageInstanceManager {
clearCache(): void;
openStorage<TStorage extends IStorage>(cls: Constructor_2<TStorage>, input: (ExplicitStorageIdentifier | DefaultStorageIdentifier) & {
- clientOpener: () => Promise<DatasetClient | KeyValueStoreClient | RequestQueueClient>;
- clientCacheKey: Hashable;
+ backendOpener: () => Promise<DatasetBackend | KeyValueStoreBackend | RequestQueueBackend>;
+ backendCacheKey: Hashable;
}): Promise<TStorage>;
removeFromCache(instance: IStorage): void;
}
@@ -1915,7 +1915,7 @@ export interface StorageOpenOptions {
config?: Configuration;
httpClient?: BaseHttpClient;
proxyConfiguration?: ProxyConfiguration;
- storageClient?: StorageClient;
+ storageBackend?: StorageBackend;
}
// @public
diff --git a/docs/public-api/crawlee-fs-storage.api.md b/docs/public-api/crawlee-fs-storage.api.md
index 321281eeb..79553b4b1 100644
--- a/docs/public-api/crawlee-fs-storage.api.md
+++ b/docs/public-api/crawlee-fs-storage.api.md
@@ -12,21 +12,21 @@ import type { FileSystemRequestQueueClient } from '@crawlee/fs-storage-native';
import type * as storage from '@crawlee/types';
// @public
-export class FileSystemStorageClient implements storage.StorageClient {
+export class FileSystemStorageBackend implements storage.StorageBackend {
constructor(options: FileSystemStorageOptions);
// (undocumented)
- createDatasetClient(options?: storage.CreateDatasetClientOptions): Promise<storage.DatasetClient>;
+ createDatasetBackend(options?: storage.CreateDatasetBackendOptions): Promise<storage.DatasetBackend>;
// (undocumented)
- createKeyValueStoreClient(options?: storage.CreateKeyValueStoreClientOptions): Promise<storage.KeyValueStoreClient>;
+ createKeyValueStoreBackend(options?: storage.CreateKeyValueStoreBackendOptions): Promise<storage.KeyValueStoreBackend>;
// (undocumented)
- createRequestQueueClient(options?: storage.CreateRequestQueueClientOptions): Promise<storage.RequestQueueClient>;
+ createRequestQueueBackend(options?: storage.CreateRequestQueueBackendOptions): Promise<storage.RequestQueueBackend>;
// (undocumented)
- readonly datasetClientCache: DatasetClient[];
+ readonly datasetBackendCache: DatasetBackend[];
// (undocumented)
readonly datasetsDirectory: string;
- getStorageClientCacheKey(): string;
+ getStorageBackendCacheKey(): string;
// (undocumented)
- readonly keyValueStoreCache: KeyValueStoreClient[];
+ readonly keyValueStoreBackendCache: KeyValueStoreBackend[];
// (undocumented)
readonly keyValueStoresDirectory: string;
// (undocumented)
@@ -37,7 +37,7 @@ export class FileSystemStorageClient implements storage.StorageClient {
// (undocumented)
readonly requestQueueAccess: 'single' | 'shared';
// (undocumented)
- readonly requestQueueCache: RequestQueueClient[];
+ readonly requestQueueBackendCache: RequestQueueBackend[];
// (undocumented)
readonly requestQueuesDirectory: string;
// (undocumented)
diff --git a/docs/public-api/crawlee-http.api.md b/docs/public-api/crawlee-http.api.md
index 9b76d3d54..3eb9c2181 100644
--- a/docs/public-api/crawlee-http.api.md
+++ b/docs/public-api/crawlee-http.api.md
@@ -153,7 +153,7 @@ export class HttpCrawler<Context extends InternalHttpCrawlingContext<any, any> =
onSkippedRequest: Predicate<Function> & BasePredicate<Function | undefined>;
httpClient: ObjectPredicate<object> & BasePredicate<object | undefined>;
configuration: ObjectPredicate<object> & BasePredicate<object | undefined>;
- storageClient: ObjectPredicate<object> & BasePredicate<object | undefined>;
+ storageBackend: ObjectPredicate<object> & BasePredicate<object | undefined>;
eventManager: ObjectPredicate<object> & BasePredicate<object | undefined>;
logger: ObjectPredicate<object> & BasePredicate<object | undefined>;
minConcurrency: NumberPredicate & BasePredicate<number | undefined>;
diff --git a/docs/public-api/crawlee-jsdom.api.md b/docs/public-api/crawlee-jsdom.api.md
index 4f88c93e8..2ecbc1013 100644
--- a/docs/public-api/crawlee-jsdom.api.md
+++ b/docs/public-api/crawlee-jsdom.api.md
@@ -93,7 +93,7 @@ export class JSDOMCrawler<ContextExtension = Dictionary<never>, ExtendedContext
onSkippedRequest: Predicate<Function> & BasePredicate<Function | undefined>;
httpClient: ObjectPredicate<object> & BasePredicate<object | undefined>;
configuration: ObjectPredicate<object> & BasePredicate<object | undefined>;
- storageClient: ObjectPredicate<object> & BasePredicate<object | undefined>;
+ storageBackend: ObjectPredicate<object> & BasePredicate<object | undefined>;
eventManager: ObjectPredicate<object> & BasePredicate<object | undefined>;
logger: ObjectPredicate<object> & BasePredicate<object | undefined>;
minConcurrency: NumberPredicate & BasePredicate<number | undefined>;
diff --git a/docs/public-api/crawlee-playwright.api.md b/docs/public-api/crawlee-playwright.api.md
index 4e8ddd47e..877395556 100644
--- a/docs/public-api/crawlee-playwright.api.md
+++ b/docs/public-api/crawlee-playwright.api.md
@@ -316,7 +316,7 @@ export class PlaywrightCrawler<ContextExtension = Dictionary_2<never>, ExtendedC
onSkippedRequest: Predicate<Function> & BasePredicate<Function | undefined>;
httpClient: ObjectPredicate<object> & BasePredicate<object | undefined>;
configuration: ObjectPredicate<object> & BasePredicate<object | undefined>;
- storageClient: ObjectPredicate<object> & BasePredicate<object | undefined>;
+ storageBackend: ObjectPredicate<object> & BasePredicate<object | undefined>;
eventManager: ObjectPredicate<object> & BasePredicate<object | undefined>;
logger: ObjectPredicate<object> & BasePredicate<object | undefined>;
minConcurrency: NumberPredicate & BasePredicate<number | undefined>;
diff --git a/docs/public-api/crawlee-puppeteer.api.md b/docs/public-api/crawlee-puppeteer.api.md
index 75c9c3bd8..d1de76588 100644
--- a/docs/public-api/crawlee-puppeteer.api.md
+++ b/docs/public-api/crawlee-puppeteer.api.md
@@ -238,7 +238,7 @@ export class PuppeteerCrawler<ContextExtension = Dictionary<never>, ExtendedCont
onSkippedRequest: Predicate<Function> & BasePredicate<Function | undefined>;
httpClient: ObjectPredicate<object> & BasePredicate<object | undefined>;
configuration: ObjectPredicate<object> & BasePredicate<object | undefined>;
- storageClient: ObjectPredicate<object> & BasePredicate<object | undefined>;
+ storageBackend: ObjectPredicate<object> & BasePredicate<object | undefined>;
eventManager: ObjectPredicate<object> & BasePredicate<object | undefined>;
logger: ObjectPredicate<object> & BasePredicate<object | undefined>;
minConcurrency: NumberPredicate & BasePredicate<number | undefined>;
diff --git a/docs/public-api/crawlee-stagehand.api.md b/docs/public-api/crawlee-stagehand.api.md
index fda999e31..8742ca6a1 100644
--- a/docs/public-api/crawlee-stagehand.api.md
+++ b/docs/public-api/crawlee-stagehand.api.md
@@ -116,7 +116,7 @@ export class StagehandCrawler<ContextExtension = Dictionary<never>, ExtendedCont
onSkippedRequest: Predicate<Function> & BasePredicate<Function | undefined>;
httpClient: ObjectPredicate<object> & BasePredicate<object | undefined>;
configuration: ObjectPredicate<object> & BasePredicate<object | undefined>;
- storageClient: ObjectPredicate<object> & BasePredicate<object | undefined>;
+ storageBackend: ObjectPredicate<object> & BasePredicate<object | undefined>;
eventManager: ObjectPredicate<object> & BasePredicate<object | undefined>;
logger: ObjectPredicate<object> & BasePredicate<object | undefined>;
minConcurrency: NumberPredicate & BasePredicate<number | undefined>;
diff --git a/docs/public-api/crawlee-types.api.md b/docs/public-api/crawlee-types.api.md
index 0290166ab..348a3491f 100644
--- a/docs/public-api/crawlee-types.api.md
+++ b/docs/public-api/crawlee-types.api.md
@@ -78,28 +78,28 @@ export interface CrawleeLoggerOptions {
}
// @public
-export type CreateDatasetClientOptions = StorageIdentifier;
+export type CreateDatasetBackendOptions = StorageIdentifier;
// @public
-export type CreateKeyValueStoreClientOptions = StorageIdentifier;
+export type CreateKeyValueStoreBackendOptions = StorageIdentifier;
// @public
-export type CreateRequestQueueClientOptions = StorageIdentifier & {
+export type CreateRequestQueueBackendOptions = StorageIdentifier & {
clientKey?: string;
timeoutSecs?: number;
};
// @public (undocumented)
-export interface DatasetClient<Data extends Dictionary = Dictionary> {
+export interface DatasetBackend<Data extends Dictionary = Dictionary> {
drop(): Promise<void>;
- getData(options?: DatasetClientListOptions): Promise<PaginatedList<Data>>;
+ getData(options?: DatasetBackendListOptions): Promise<PaginatedList<Data>>;
getMetadata(): Promise<DatasetInfo>;
purge(): Promise<void>;
pushData(items: Data[]): Promise<void>;
}
// @public (undocumented)
-export interface DatasetClientListOptions {
+export interface DatasetBackendListOptions {
// (undocumented)
desc?: boolean;
// (undocumented)
@@ -208,7 +208,7 @@ export interface ISessionPool {
}
// @public
-export interface KeyValueStoreClient {
+export interface KeyValueStoreBackend {
deleteValue(key: string): Promise<void>;
drop(): Promise<void>;
getMetadata(): Promise<KeyValueStoreInfo>;
@@ -340,7 +340,7 @@ export type RedirectHandler = (redirectResponse: Response, updatedRequest: {
}) => void;
// @public
-export interface RequestQueueClient {
+export interface RequestQueueBackend {
addBatchOfRequests(requests: RequestSchema[], options?: RequestQueueOperationOptions): Promise<BatchAddRequestsResult>;
drop(): Promise<void>;
fetchNextRequest(): Promise<UpdateRequestSchema | undefined>;
@@ -465,11 +465,11 @@ export interface SetStatusMessageOptions {
}
// @public
-export interface StorageClient {
- createDatasetClient(options?: CreateDatasetClientOptions): Promise<DatasetClient>;
- createKeyValueStoreClient(options?: CreateKeyValueStoreClientOptions): Promise<KeyValueStoreClient>;
- createRequestQueueClient(options?: CreateRequestQueueClientOptions): Promise<RequestQueueClient>;
- getStorageClientCacheKey?(): string;
+export interface StorageBackend {
+ createDatasetBackend(options?: CreateDatasetBackendOptions): Promise<DatasetBackend>;
+ createKeyValueStoreBackend(options?: CreateKeyValueStoreBackendOptions): Promise<KeyValueStoreBackend>;
+ createRequestQueueBackend(options?: CreateRequestQueueBackendOptions): Promise<RequestQueueBackend>;
+ getStorageBackendCacheKey?(): string;
// (undocumented)
purge?(): Promise<void>;
// (undocumented)
diff --git a/docs/upgrading/upgrading_v3.md b/docs/upgrading/upgrading_v3.md
index 39b6091c9..d61fc560a 100644
--- a/docs/upgrading/upgrading_v3.md
+++ b/docs/upgrading/upgrading_v3.md
@@ -31,7 +31,7 @@ The [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of sever
- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative
- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package
- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods
-- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)
+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/3.16/core/interface/StorageClient)
### Installing Crawlee
diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.md
index a1ed70767..e0a671f69 100644
--- a/docs/upgrading/upgrading_v4.md
+++ b/docs/upgrading/upgrading_v4.md
@@ -344,7 +344,7 @@ The crawling context in the `FileDownload` crawler no longer includes the `body`
## `KeyValueStore.getPublicUrl` is now async
-The `KeyValueStore.getPublicUrl` method is now asynchronous and reads the public URL directly from the storage client.
+The `KeyValueStore.getPublicUrl` method is now asynchronous and reads the public URL directly from the storage backend.
## `preNavigationHooks` in `HttpCrawler` no longer accepts `gotOptions` object
@@ -392,9 +392,9 @@ The service management functionality has been extracted from `Configuration` int
The following methods and properties have been removed from `Configuration`:
-- `Configuration.getStorageClient()` - moved to `ServiceLocator.getStorageClient()`
+- `Configuration.getStorageClient()` - moved to `ServiceLocator.getStorageBackend()`
- `Configuration.getEventManager()` - moved to `ServiceLocator.getEventManager()`
-- `Configuration.useStorageClient()` - use `ServiceLocator.setStorageClient()` instead
+- `Configuration.useStorageClient()` - use `ServiceLocator.setStorageBackend()` instead
- `Configuration.useEventManager()` - use `ServiceLocator.setEventManager()` instead
- `Configuration.resetGlobalState()` - use `serviceLocator.reset()` instead
- `Configuration.storageManagers` - moved to `ServiceLocator.storageManagers`
@@ -410,34 +410,35 @@ If you were using the removed `Configuration` methods directly, you need to upda
import { Configuration } from 'crawlee';
const config = Configuration.getGlobalConfig();
-const storageClient = config.getStorageClient();
+const storageBackend = config.getStorageClient();
const eventManager = config.getEventManager();
// or static methods
-const storageClient = Configuration.getStorageClient();
+const storageBackend = Configuration.getStorageClient();
+// (both of these are the removed v3 APIs)
```
**After:**
```typescript
import { serviceLocator } from 'crawlee';
-const storageClient = serviceLocator.getStorageClient();
+const storageBackend = serviceLocator.getStorageBackend();
const eventManager = serviceLocator.getEventManager();
```
### Using per-crawler services (recommended)
-The new `ServiceLocator` supports per-crawler service isolation, allowing you to use different storage clients or event managers for different crawlers by passing them via options:
+The new `ServiceLocator` supports per-crawler service isolation, allowing you to use different storage backends or event managers for different crawlers by passing them via options:
```typescript
-import { BasicCrawler, Configuration, LocalEventManager, MemoryStorageClient } from 'crawlee';
+import { BasicCrawler, Configuration, LocalEventManager, MemoryStorageBackend } from 'crawlee';
const crawler = new BasicCrawler({
requestHandler: async ({ request, log }) => {
log.info(`Processing ${request.url}`);
},
configuration: new Configuration({ headless: false }),
- storageClient: new MemoryStorageClient(),
+ storageBackend: new MemoryStorageBackend(),
eventManager: LocalEventManager.fromConfig(),
});
@@ -449,10 +450,10 @@ await crawler.run(['https://example.com']);
For most use cases, the global `serviceLocator` singleton works well:
```typescript
-import { serviceLocator, BasicCrawler, MemoryStorageClient } from 'crawlee';
+import { serviceLocator, BasicCrawler, MemoryStorageBackend } from 'crawlee';
// Configure global services (optional)
-serviceLocator.setStorageClient(new MemoryStorageClient());
+serviceLocator.setStorageBackend(new MemoryStorageBackend());
// All crawlers will use the global service locator by default
const crawler = new BasicCrawler({
@@ -506,9 +507,9 @@ The `persistCookiesPerSession` crawler option has been renamed to `saveResponseC
Several internal Crawlee keys were prefixed with the `SDK_` prefix for legacy reasons - these keys now start with `CRAWLEE_` instead. These are, e.g., `CRAWLEE_SESSION_POOL_STATE` or `CRAWLEE_CRAWLER_STATISTICS_{n}`.
-## `StorageClient` interface simplified
+## `StorageBackend` interface simplified
-The `StorageClient` interface (from `@crawlee/types`) has been redesigned to match the simplified architecture from Crawlee for Python. A new storage backend now needs **4 classes** instead of the previous 7.
+The `StorageBackend` interface (from `@crawlee/types`, formerly named `StorageClient`) has been redesigned to match the simplified architecture from Crawlee for Python. A new storage backend now needs **4 classes** instead of the previous 7.
### What changed
@@ -520,18 +521,18 @@ The three **collection client** interfaces have been removed:
Along with their associated types (`DatasetCollectionData`, `DatasetCollectionClientOptions`, and the `Dataset` interface from `@crawlee/types`).
-The `StorageClient` interface changed from synchronous sub-client getters to **async factory methods**:
+The `StorageBackend` interface changed from synchronous sub-client getters to **async factory methods**:
| Before (v3) | After (v4) |
|---|---|
-| `client.dataset(id)` | `client.createDatasetClient({ id?, name? })` |
-| `client.datasets().getOrCreate(name)` | _(absorbed into `createDatasetClient`)_ |
-| `client.keyValueStore(id)` | `client.createKeyValueStoreClient({ id?, name? })` |
-| `client.keyValueStores().getOrCreate(name)` | _(absorbed into `createKeyValueStoreClient`)_ |
-| `client.requestQueue(id, opts)` | `client.createRequestQueueClient({ id?, name?, clientKey?, timeoutSecs? })` |
-| `client.requestQueues().getOrCreate(name)` | _(absorbed into `createRequestQueueClient`)_ |
+| `client.dataset(id)` | `backend.createDatasetBackend({ id?, name? })` |
+| `client.datasets().getOrCreate(name)` | _(absorbed into `createDatasetBackend`)_ |
+| `client.keyValueStore(id)` | `backend.createKeyValueStoreBackend({ id?, name? })` |
+| `client.keyValueStores().getOrCreate(name)` | _(absorbed into `createKeyValueStoreBackend`)_ |
+| `client.requestQueue(id, opts)` | `backend.createRequestQueueBackend({ id?, name?, clientKey?, timeoutSecs? })` |
+| `client.requestQueues().getOrCreate(name)` | _(absorbed into `createRequestQueueBackend`)_ |
-The sub-client interfaces (`DatasetClient`, `KeyValueStoreClient`, `RequestQueueClient`) have been aligned with their Python counterparts:
+The sub-backend interfaces (`DatasetBackend`, `KeyValueStoreBackend`, `RequestQueueBackend`, formerly `DatasetClient` / `KeyValueStoreClient` / `RequestQueueClient`) have been aligned with their Python counterparts:
| Before (v3) | After (v4) |
|---|---|
@@ -540,7 +541,7 @@ The sub-client interfaces (`DatasetClient`, `KeyValueStoreClient`, `RequestQueue
| `delete()` | `drop()` |
| _(n/a)_ | `purge()` (new — clears data, keeps storage) |
-**`DatasetClient`:**
+**`DatasetBackend`:**
| Before (v3) | After (v4) |
|---|---|
@@ -549,7 +550,7 @@ The sub-client interfaces (`DatasetClient`, `KeyValueStoreClient`, `RequestQueue
| `listEntries(options?)` | Removed (handled by `Dataset` frontend) |
| `downloadItems()` | Removed |
-**`KeyValueStoreClient`:**
+**`KeyValueStoreBackend`:**
| Before (v3) | After (v4) |
|---|---|
@@ -560,9 +561,9 @@ The sub-client interfaces (`DatasetClient`, `KeyValueStoreClient`, `RequestQueue
| `listKeys(options?)` → `KeyValueStoreClientListData` | `listKeys(options?)` → `KeyValueStoreListKeysResult` (a single self-describing page) |
| `keys()`, `values()`, `entries()` | Removed (handled by `KeyValueStore` frontend) |
-**`RequestQueueClient`:**
+**`RequestQueueBackend`:**
-The request queue client was reduced from 12 methods to 10. The distributed-locking protocol (`listAndLockHead` → `prolongRequestLock` → `deleteRequestLock`) and the queue-head/consistency bookkeeping that used to live in the `RequestQueue` frontend have been removed from the interface; coordinating multiple clients accessing the same queue (e.g. request locking on the Apify platform) is now an internal concern of the client implementation.
+The request queue backend was reduced from 12 methods to 10. The distributed-locking protocol (`listAndLockHead` → `prolongRequestLock` → `deleteRequestLock`) and the queue-head/consistency bookkeeping that used to live in the `RequestQueue` frontend have been removed from the interface; coordinating multiple clients accessing the same queue (e.g. request locking on the Apify platform) is now an internal concern of the backend implementation.
| Before (v3) | After (v4) |
|---|---|
@@ -580,14 +581,14 @@ The request queue client was reduced from 12 methods to 10. The distributed-lock
The lifecycle is now: `fetchNextRequest()` hands out a pending request and marks it in progress; once processed, call `markRequestAsHandled(request)`; on failure call `reclaimRequest(request, { forefront? })` to return it to the queue.
-Methods that may have "nothing" to return now consistently resolve to `undefined` rather than `null`. `fetchNextRequest()` resolves to `undefined` when there is nothing to fetch, and `markRequestAsHandled()` / `reclaimRequest()` resolve to `undefined` when the request is not something the client is currently processing (a no-op, not an error). This matches the `undefined` already returned by `getRequest()`, `KeyValueStoreClient.getValue()`, and `getPublicUrl()`, so the whole client family uses a single "absent" sentinel. If you implemented a custom client that returned `null` from these methods, return `undefined` instead.
+Methods that may have "nothing" to return now consistently resolve to `undefined` rather than `null`. `fetchNextRequest()` resolves to `undefined` when there is nothing to fetch, and `markRequestAsHandled()` / `reclaimRequest()` resolve to `undefined` when the request is not something the backend is currently processing (a no-op, not an error). This matches the `undefined` already returned by `getRequest()`, `KeyValueStoreBackend.getValue()`, and `getPublicUrl()`, so the whole backend family uses a single "absent" sentinel. If you implemented a custom backend that returned `null` from these methods, return `undefined` instead.
-`RequestQueueClient.isEmpty()` and `RequestQueueClient.isFinished()` answer two different questions:
+`RequestQueueBackend.isEmpty()` and `RequestQueueBackend.isFinished()` answer two different questions:
- `isEmpty()` is the weak check — `true` when the next `fetchNextRequest()` would return `undefined`, i.e. there is nothing left to fetch right now. Requests that are currently in progress (fetched but not yet handled or reclaimed) are **not** counted, because they are not fetchable. This is what drives the crawler's task scheduling.
- `isFinished()` is the strong check — `true` only when there are no pending requests **and** no requests currently in progress (including those locked by other clients sharing the queue). This is what determines whether crawling is actually done. An in-progress request keeps the queue *empty but not finished*, which is what stops a crawler from shutting down while a request is still being processed.
-The separate `RequestQueueV1`/`RequestQueueV2` classes (and the `RequestProvider` base class) have been removed. They no longer differ in behavior — request coordination is now internal to the storage client — so they are merged into a single `RequestQueue` class. Replace any `RequestQueueV1`, `RequestQueueV2`, or `RequestProvider` imports with `RequestQueue`.
+The separate `RequestQueueV1`/`RequestQueueV2` classes (and the `RequestProvider` base class) have been removed. They no longer differ in behavior — request coordination is now internal to the storage backend — so they are merged into a single `RequestQueue` class. Replace any `RequestQueueV1`, `RequestQueueV2`, or `RequestProvider` imports with `RequestQueue`. (Request coordination is now internal to the storage backend.)
The `requestLocking` crawler experiment has been removed, along with the `experiments` crawler option and the `CrawlerExperiments` type that contained it. Request locking has been the default since v3.10 and there is no longer an alternative implementation to opt out to, so the flag did nothing. Delete any `experiments: { requestLocking: ... }` from your crawler options:
@@ -602,7 +603,7 @@ The `requestLocking` crawler experiment has been removed, along with the `experi
});
```
-The `RequestQueue.requestLockSecs` property has been removed. Because request locking is now internal to the storage client, the lock duration is no longer configured on the queue. When you run a crawler, it automatically tells the queue how long it expects to hold a request (based on `requestHandlerTimeoutMillis`), so a long-running request handler will not have its request handed out a second time — you usually don't need to configure anything.
+The `RequestQueue.requestLockSecs` property has been removed. Because request locking is now internal to the storage backend, the lock duration is no longer configured on the queue. When you run a crawler, it automatically tells the queue how long it expects to hold a request (based on `requestHandlerTimeoutMillis`), so a long-running request handler will not have its request handed out a second time — you usually don't need to configure anything.
If you use a `RequestQueue` outside of a crawler and your processing may exceed the 3-minute default lock, call `setExpectedRequestProcessingTimeSecs(secs)` on the queue to raise it:
@@ -613,13 +614,13 @@ const queue = await RequestQueue.open();
queue.setExpectedRequestProcessingTimeSecs(600);
```
-The `RequestQueue.internalTimeoutMillis` property and the associated "stuck queue" self-recovery have been removed. In v3 the `RequestQueue` frontend kept its own copy of the queue head and in-progress set, which could drift out of sync with the backing storage (an eventual-consistency hazard on the Apify platform); `isFinished()` watched for inactivity exceeding `internalTimeoutMillis` and reset that frontend state to recover. In v4 the frontend no longer holds any such bookkeeping — the storage client is the single source of truth — so there is nothing for a reset to fix, and stuck request locks now self-heal on expiry. Any consistency-recovery logic that is genuinely specific to the Apify platform's distributed storage belongs in the Apify SDK's client implementation instead, and is tracked in [apify/crawlee#3328](https://github.com/apify/crawlee/issues/3328).
+The `RequestQueue.internalTimeoutMillis` property and the associated "stuck queue" self-recovery have been removed. In v3 the `RequestQueue` frontend kept its own copy of the queue head and in-progress set, which could drift out of sync with the backing storage (an eventual-consistency hazard on the Apify platform); `isFinished()` watched for inactivity exceeding `internalTimeoutMillis` and reset that frontend state to recover. In v4 the frontend no longer holds any such bookkeeping — the storage backend is the single source of truth — so there is nothing for a reset to fix, and stuck request locks now self-heal on expiry. Any consistency-recovery logic that is genuinely specific to the Apify platform's distributed storage belongs in the Apify SDK's client implementation instead, and is tracked in [apify/crawlee#3328](https://github.com/apify/crawlee/issues/3328).
-**Apify-specific fields removed from storage metadata.** The metadata returned by `getMetadata()` (`DatasetInfo`, `KeyValueStoreInfo`, `RequestQueueInfo`) has been trimmed to what is meaningful for any storage backend. The following platform-specific fields were dropped: `actId`, `actRunId`, `userId`, and — on `RequestQueueInfo` — `expireAt` and `hadMultipleClients`. The per-storage `stats` field (and its `DatasetStats` / `KeyValueStoreStats` / `RequestQueueStats` types) was removed as well. If you consumed any of these, read them from the Apify API client directly; a custom `StorageClient` should simply stop returning them.
+**Apify-specific fields removed from storage metadata.** The metadata returned by `getMetadata()` (`DatasetInfo`, `KeyValueStoreInfo`, `RequestQueueInfo`) has been trimmed to what is meaningful for any storage backend. The following platform-specific fields were dropped: `actId`, `actRunId`, `userId`, and — on `RequestQueueInfo` — `expireAt` and `hadMultipleClients`. The per-storage `stats` field (and its `DatasetStats` / `KeyValueStoreStats` / `RequestQueueStats` types) was removed as well. If you consumed any of these, read them from the Apify API client directly; a custom `StorageBackend` should simply stop returning them.
**Removed types** from `@crawlee/types`: `DatasetClientUpdateOptions`, `KeyValueStoreClientUpdateOptions`, `KeyValueStoreRecordOptions`, `KeyValueStoreClientListData`, `KeyValueStoreClientGetRecordOptions`, `QueueHead`, `RequestQueueHeadItem`, `ListOptions`, `ListAndLockOptions`, `ListAndLockHeadResult`, `ProlongRequestLockOptions`, `ProlongRequestLockResult`, `DeleteRequestLockOptions`, `DatasetStats`, `KeyValueStoreStats`, `RequestQueueStats`. `KeyValueStoreClientListOptions` was renamed to `KeyValueStoreListKeysOptions`.
-The high-level storage classes (`Dataset`, `KeyValueStore`, `RequestQueue`) now receive their sub-client directly in the constructor options instead of receiving a `StorageClient` and calling its methods.
+The high-level storage classes (`Dataset`, `KeyValueStore`, `RequestQueue`) now receive their sub-backend directly in the constructor options (via the `backend` option) instead of receiving a `StorageBackend` and calling its methods.
### `RecordOptions` simplified
@@ -627,7 +628,7 @@ The high-level storage classes (`Dataset`, `KeyValueStore`, `RequestQueue`) now
### `maybeStringify` is removed
-The `maybeStringify` helper exported from `@crawlee/core` has been removed. Value (de)serialization now lives entirely in the `KeyValueStore` frontend: writing serializes the value (and infers its content type), reading parses it back, and the storage client is a plain byte transport. If you imported `maybeStringify` directly, use the `serializeValue` / `parseValue` functions exported from `@crawlee/core` instead.
+The `maybeStringify` helper exported from `@crawlee/core` has been removed. Value (de)serialization now lives entirely in the `KeyValueStore` frontend: writing serializes the value (and infers its content type), reading parses it back, and the storage backend is a plain byte transport. If you imported `maybeStringify` directly, use the `serializeValue` / `parseValue` functions exported from `@crawlee/core` instead.
### `KeyValueStoreIteratorOptions` simplified
@@ -647,76 +648,76 @@ The `list()` method on collection clients (e.g. `client.datasets().list()`) has
### Migration guide
-If you implemented a custom `StorageClient`, you need to:
+If you implemented a custom `StorageBackend`, you need to:
1. Remove your `*CollectionClient` classes.
-2. Replace the six getter methods (`dataset`, `datasets`, `keyValueStore`, `keyValueStores`, `requestQueue`, `requestQueues`) with three async factory methods (`createDatasetClient`, `createKeyValueStoreClient`, `createRequestQueueClient`). Each factory should handle both opening an existing storage and creating a new one.
-3. Apply the sub-client renames listed above (`get` → `getMetadata`, `delete` → `drop`, etc.) and implement the new `purge()` method.
+2. Replace the six getter methods (`dataset`, `datasets`, `keyValueStore`, `keyValueStores`, `requestQueue`, `requestQueues`) with three async factory methods (`createDatasetBackend`, `createKeyValueStoreBackend`, `createRequestQueueBackend`). Each factory should handle both opening an existing storage and creating a new one.
+3. Apply the sub-backend renames listed above (`get` → `getMetadata`, `delete` → `drop`, etc.) and implement the new `purge()` method.
-## `MemoryStorage` split into `FileSystemStorageClient` and `MemoryStorageClient`
+## `MemoryStorage` split into `FileSystemStorageBackend` and `MemoryStorageBackend`
-In v3, the single `MemoryStorage` class from `@crawlee/memory-storage` did double duty: it kept everything in memory *and*, by default, mirrored it to disk (toggled via the `persistStorage` option / `CRAWLEE_PERSIST_STORAGE` environment variable). In v4 these two responsibilities are split into two independent classes, and the default storage client now persists to disk.
+In v3, the single `MemoryStorage` class from `@crawlee/memory-storage` did double duty: it kept everything in memory *and*, by default, mirrored it to disk (toggled via the `persistStorage` option / `CRAWLEE_PERSIST_STORAGE` environment variable). In v4 these two responsibilities are split into two independent classes, and the default storage backend now persists to disk.
-- **`FileSystemStorageClient`** (new, in the new `@crawlee/fs-storage` package) — always persists storage to the local directory (`CRAWLEE_STORAGE_DIR`, default `./storage`). This is what you get implicitly when you don't configure a storage client, and it is the behavior the old `MemoryStorage` had with its default `persistStorage: true`.
-- **`MemoryStorageClient`** (the renamed `MemoryStorage`, now part of `@crawlee/core`) — keeps everything purely in memory and **never touches the disk**. This matches the old `MemoryStorage` with `persistStorage: false`. The standalone `@crawlee/memory-storage` package no longer exists; its code was merged into `@crawlee/core`.
+- **`FileSystemStorageBackend`** (new, in the new `@crawlee/fs-storage` package) — always persists storage to the local directory (`CRAWLEE_STORAGE_DIR`, default `./storage`). This is what you get implicitly when you don't configure a storage backend, and it is the behavior the old `MemoryStorage` had with its default `persistStorage: true`.
+- **`MemoryStorageBackend`** (the renamed `MemoryStorage`, now part of `@crawlee/core`) — keeps everything purely in memory and **never touches the disk**. This matches the old `MemoryStorage` with `persistStorage: false`. The standalone `@crawlee/memory-storage` package no longer exists; its code was merged into `@crawlee/core`.
Both classes are re-exported from the `crawlee` meta-package.
-### The default storage client now persists to disk
+### The default storage backend now persists to disk
-Which client backs the implicit default is decided by `Configuration.persistStorage` (still controllable via the `CRAWLEE_PERSIST_STORAGE` environment variable): `true` (the default) selects `FileSystemStorageClient`, `false` selects `MemoryStorageClient`. If you relied on the default and never set `persistStorage`, your storage is persisted to disk exactly as before — no change.
+Which client backs the implicit default is decided by `Configuration.persistStorage` (still controllable via the `CRAWLEE_PERSIST_STORAGE` environment variable): `true` (the default) selects `FileSystemStorageBackend`, `false` selects `MemoryStorageBackend`. If you relied on the default and never set `persistStorage`, your storage is persisted to disk exactly as before — no change.
### `MemoryStorage` is renamed and is now memory-only
-If you constructed the storage client explicitly, two things changed:
+If you constructed the storage backend explicitly, two things changed:
-1. **The class is renamed** `MemoryStorage` → `MemoryStorageClient`.
-2. **It no longer writes to disk.** A bare `new MemoryStorage()` in v3 persisted to disk by default; `new MemoryStorageClient()` in v4 does not. If you want persistence, use `FileSystemStorageClient` instead.
+1. **The class is renamed** `MemoryStorage` → `MemoryStorageBackend`.
+2. **It no longer writes to disk.** A bare `new MemoryStorage()` in v3 persisted to disk by default; `new MemoryStorageBackend()` in v4 does not. If you want persistence, use `FileSystemStorageBackend` instead.
**Before:**
```typescript
import { MemoryStorage } from '@crawlee/memory-storage';
// Persisted to disk by default in v3.
-const storageClient = new MemoryStorage();
+const storageBackend = new MemoryStorage();
```
**After:**
```typescript
-import { FileSystemStorageClient } from '@crawlee/fs-storage';
-import { MemoryStorageClient } from '@crawlee/core';
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
+import { MemoryStorageBackend } from '@crawlee/core';
// Persists to disk (the old default behavior):
-const storageClient = new FileSystemStorageClient({ localDataDirectory: './storage' });
+const storageBackend = new FileSystemStorageBackend({ localDataDirectory: './storage' });
// Or keep everything in memory only (the old `persistStorage: false`):
-const inMemory = new MemoryStorageClient();
+const inMemory = new MemoryStorageBackend();
```
-`MemoryStorageClient` no longer takes the `localDataDirectory`, `persistStorage`, or `writeMetadata` options — in-memory storage has nowhere to write, so they had no meaning. `FileSystemStorageClient` honors `localDataDirectory`; it always persists, so it has no `persistStorage` option, and the `writeMetadata` option has been removed there too (see [`writeMetadata` option removed](#writemetadata-option-removed)).
+`MemoryStorageBackend` no longer takes the `localDataDirectory`, `persistStorage`, or `writeMetadata` options — in-memory storage has nowhere to write, so they had no meaning. `FileSystemStorageBackend` honors `localDataDirectory`; it always persists, so it has no `persistStorage` option, and the `writeMetadata` option has been removed there too (see [`writeMetadata` option removed](#writemetadata-option-removed)).
-### No request lock expiry in `MemoryStorageClient`
+### No request lock expiry in `MemoryStorageBackend`
-Because the in-memory queue lives entirely within a single process and is never shared with another consumer, `MemoryStorageClient`'s request queue no longer uses an expiring, cross-process lock. A fetched request simply stays *in progress* until it is handled or reclaimed; it never becomes fetchable again on its own after a timeout. `setExpectedRequestProcessingTimeSecs()` is therefore a no-op for in-memory storage. (Disk-backed `FileSystemStorageClient` keeps the lock-with-expiry behavior.)
+Because the in-memory queue lives entirely within a single process and is never shared with another consumer, `MemoryStorageBackend`'s request queue no longer uses an expiring, cross-process lock. A fetched request simply stays *in progress* until it is handled or reclaimed; it never becomes fetchable again on its own after a timeout. `setExpectedRequestProcessingTimeSecs()` is therefore a no-op for in-memory storage. (Disk-backed `FileSystemStorageBackend` keeps the lock-with-expiry behavior.)
### `writeMetadata` option removed
-`FileSystemStorageClient` no longer accepts the `writeMetadata` option. The underlying file-system storage now always writes metadata files (`__metadata__.json` for each storage and a `<key>.__metadata__.json` sidecar for each key-value record), so the toggle no longer had any effect. Remove it from your storage client options:
+`FileSystemStorageBackend` no longer accepts the `writeMetadata` option. The underlying file-system storage now always writes metadata files (`__metadata__.json` for each storage and a `<key>.__metadata__.json` sidecar for each key-value record), so the toggle no longer had any effect. Remove it from your storage backend options:
```diff
- import { FileSystemStorageClient } from '@crawlee/fs-storage';
+ import { FileSystemStorageBackend } from '@crawlee/fs-storage';
- const storageClient = new FileSystemStorageClient({
+ const storageBackend = new FileSystemStorageBackend({
localDataDirectory: './storage',
- writeMetadata: true,
});
```
-`MemoryStorageClient` never accepted `writeMetadata` (it has no on-disk format to begin with), so there is nothing to change there.
+`MemoryStorageBackend` never accepted `writeMetadata` (it has no on-disk format to begin with), so there is nothing to change there.
### Out-of-band key-value files (e.g. a hand-placed `INPUT.json`)
-`FileSystemStorageClient` only fully tracks records it wrote itself (those have a `<key>.__metadata__.json` sidecar). It still reads a value file placed in the store directory out-of-band — such as a hand-written or platform-provided `INPUT.json` — by probing the requested key plus the `.json` and `.txt` extensions. A few behaviors around these "bare" files changed in v4:
+`FileSystemStorageBackend` only fully tracks records it wrote itself (those have a `<key>.__metadata__.json` sidecar). It still reads a value file placed in the store directory out-of-band — such as a hand-written or platform-provided `INPUT.json` — by probing the requested key plus the `.json` and `.txt` extensions. A few behaviors around these "bare" files changed in v4:
- **Extensionless bare files report `application/octet-stream`.** In v3 a bare value file with no extension was read as `text/plain`. In v4 the client is a plain byte transport and only infers a content type from a real extension, so an extensionless file now comes back as `application/octet-stream`. Give the file a `.json` or `.txt` extension if you need a more specific type.
- **Malformed bare files are no longer silently swallowed.** In v3 a bare `INPUT.json` containing invalid JSON was treated as a missing record (`getValue` returned `undefined`). In v4 the raw bytes are returned verbatim and parsing happens in the `KeyValueStore` frontend, so a malformed value now surfaces a parse error at read time instead of looking absent.
@@ -831,7 +832,7 @@ The harmonized loader interface differs from the old `IRequestList` in a few way
`RequestList.length()` and `RequestList.handledCount()` (and their `SitemapRequestLoader` counterparts) were renamed to `getTotalCount()` and `getHandledCount()` and are now `async` — `await` them.
-`markRequestHandled()` was renamed to `markRequestAsHandled()` across the loader and manager interfaces (`RequestList`, `SitemapRequestLoader`, `RequestQueue`, `RequestManagerTandem`) to match the storage client method of the same name (and the Python `mark_request_as_handled`). Rename any calls accordingly.
+`markRequestHandled()` was renamed to `markRequestAsHandled()` across the loader and manager interfaces (`RequestList`, `SitemapRequestLoader`, `RequestQueue`, `RequestManagerTandem`) to match the storage backend method of the same name (and the Python `mark_request_as_handled`). Rename any calls accordingly.
**Before:**
```typescript
diff --git a/packages/basic-crawler/src/internals/basic-crawler.ts b/packages/basic-crawler/src/internals/basic-crawler.ts
index 408a105a2..f01045b11 100644
--- a/packages/basic-crawler/src/internals/basic-crawler.ts
+++ b/packages/basic-crawler/src/internals/basic-crawler.ts
@@ -75,7 +75,7 @@ import type {
ISessionPool,
ProxyInfo,
SetStatusMessageOptions,
- StorageClient,
+ StorageBackend,
} from '@crawlee/types';
import { getObjectType, isAsyncIterable, isIterable, RobotsTxtFile, ROTATE_PROXY_ERRORS, sleep } from '@crawlee/utils';
import { stringify } from 'csv-stringify/sync';
@@ -421,10 +421,10 @@ export interface BasicCrawlerOptions<
configuration?: Configuration;
/**
- * Custom storage client to use for this crawler.
+ * Custom storage backend to use for this crawler.
* If provided, the crawler will use its own ServiceLocator instance instead of the global one.
*/
- storageClient?: StorageClient;
+ storageBackend?: StorageBackend;
/**
* Custom event manager to use for this crawler.
@@ -732,7 +732,7 @@ export class BasicCrawler<
httpClient: ow.optional.object,
configuration: ow.optional.object,
- storageClient: ow.optional.object,
+ storageBackend: ow.optional.object,
eventManager: ow.optional.object,
logger: ow.optional.object,
@@ -776,7 +776,7 @@ export class BasicCrawler<
// Service locator options
configuration,
- storageClient,
+ storageBackend,
eventManager,
logger,
@@ -809,12 +809,12 @@ export class BasicCrawler<
let serviceLocatorScope = { enterScope: () => {}, exitScope: () => {} };
if (
- storageClient ||
+ storageBackend ||
eventManager ||
logger ||
(configuration !== undefined && configuration !== serviceLocator.getConfiguration())
) {
- const scopedServiceLocator = new ServiceLocator(configuration, eventManager, storageClient, logger);
+ const scopedServiceLocator = new ServiceLocator(configuration, eventManager, storageBackend, logger);
serviceLocatorScope = bindMethodsToServiceLocator(scopedServiceLocator, this);
}
@@ -1344,7 +1344,7 @@ export class BasicCrawler<
await purgeDefaultStorages({
onlyPurgeOnce: true,
- client: serviceLocator.getStorageClient(),
+ storageBackend: serviceLocator.getStorageBackend(),
config: serviceLocator.getConfiguration(),
});
@@ -1404,7 +1404,7 @@ export class BasicCrawler<
});
}
- const client = serviceLocator.getStorageClient();
+ const client = serviceLocator.getStorageBackend();
if (client.teardown) {
let finished = false;
@@ -1498,7 +1498,7 @@ export class BasicCrawler<
/**
* Tells a request manager how long we expect to hold a fetched request, so that one backed by a
- * locking storage client keeps it reserved for slightly longer than the request handler timeout
+ * locking storage backend keeps it reserved for slightly longer than the request handler timeout
* (with some padding for overhead), but never for less than a minute. This prevents a long-running
* request from being handed out a second time while it is still being processed — and it works
* regardless of whether the manager is a plain {@apilink RequestQueue} or a `RequestManagerTandem`.
@@ -2018,7 +2018,7 @@ export class BasicCrawler<
} finally {
// Safety net - return the request to the queue if nobody managed to mark it as handled
// or reclaim it before (e.g. after a CriticalError). Reclaiming a request that is no longer
- // in progress is a harmless no-op on the storage client.
+ // in progress is a harmless no-op on the storage backend.
if (isRequestLocked && requestSource instanceof RequestQueue) {
try {
await requestSource.reclaimRequest(request);
diff --git a/packages/basic-crawler/test/batch-add-requests.test.ts b/packages/basic-crawler/test/batch-add-requests.test.ts
index 827e17014..18ee2070a 100644
--- a/packages/basic-crawler/test/batch-add-requests.test.ts
+++ b/packages/basic-crawler/test/batch-add-requests.test.ts
@@ -1,10 +1,10 @@
import { BasicCrawler } from '@crawlee/basic';
-import { MemoryStorageClient, serviceLocator, SessionPool } from '@crawlee/core';
+import { MemoryStorageBackend, serviceLocator, SessionPool } from '@crawlee/core';
describe('BasicCrawler#addRequests with big batch sizes', () => {
beforeEach(async () => {
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
});
const requestTemplates = Array.from({ length: 2000 }, (_, i) => ({ url: `https://example.com/${i}` }));
@@ -58,7 +58,7 @@ describe('BasicCrawler#addRequests with big batch sizes', () => {
describe('BasicCrawler - request.sessionId', () => {
beforeEach(async () => {
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
});
test('uses the session matching request.sessionId from the session pool', async () => {
diff --git a/packages/core/README.md b/packages/core/README.md
index 167cf6112..0be6b71ac 100644
--- a/packages/core/README.md
+++ b/packages/core/README.md
@@ -16,7 +16,7 @@ The [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of sever
- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative
- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package
- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods
-- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)
+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageBackend`](https://crawlee.dev/js/api/core/interface/StorageBackend)
## Installing Crawlee
diff --git a/packages/core/src/autoscaling/client_load_signal.ts b/packages/core/src/autoscaling/client_load_signal.ts
index 037f44c52..d5d92f0a5 100644
--- a/packages/core/src/autoscaling/client_load_signal.ts
+++ b/packages/core/src/autoscaling/client_load_signal.ts
@@ -1,4 +1,4 @@
-import type { StorageClient } from '@crawlee/types';
+import type { StorageBackend } from '@crawlee/types';
import type { LoadSnapshot } from './load_signal.js';
import { SnapshotStore } from './load_signal.js';
@@ -10,7 +10,7 @@ export interface ClientSnapshot extends LoadSnapshot {
}
export interface ClientLoadSignalOptions {
- client: StorageClient;
+ client: StorageBackend;
clientSnapshotIntervalSecs?: number;
maxClientErrors?: number;
overloadedRatio?: number;
diff --git a/packages/core/src/autoscaling/snapshotter.ts b/packages/core/src/autoscaling/snapshotter.ts
index c6c187652..4e34552ee 100644
--- a/packages/core/src/autoscaling/snapshotter.ts
+++ b/packages/core/src/autoscaling/snapshotter.ts
@@ -1,4 +1,4 @@
-import type { StorageClient } from '@crawlee/types';
+import type { StorageBackend } from '@crawlee/types';
import ow from 'ow';
import type { Configuration } from '../configuration.js';
@@ -61,7 +61,7 @@ export interface SnapshotterOptions {
log?: CrawleeLogger;
/** @internal */
- client?: StorageClient;
+ client?: StorageBackend;
/** @internal */
config?: Configuration;
@@ -95,7 +95,7 @@ export interface SnapshotterOptions {
*/
export class Snapshotter {
log: CrawleeLogger;
- client: StorageClient;
+ client: StorageBackend;
config: Configuration;
private readonly memorySignal: MemoryLoadSignal;
@@ -156,7 +156,7 @@ export class Snapshotter {
maxClientErrors = 3,
log = serviceLocator.getLogger(),
config = serviceLocator.getConfiguration(),
- client = serviceLocator.getStorageClient(),
+ client = serviceLocator.getStorageBackend(),
} = options;
this.log = log.child({ prefix: 'Snapshotter' });
diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts
index 5be7d9575..ee84b93af 100644
--- a/packages/core/src/index.ts
+++ b/packages/core/src/index.ts
@@ -17,4 +17,4 @@ export * from './validators.js';
export * from './cookie_utils.js';
export * from './recoverable_state.js';
export { PseudoUrl } from '@apify/pseudo_url';
-export type { Dictionary, Awaitable, Constructor, StorageClient, Cookie, QueueOperationInfo } from '@crawlee/types';
+export type { Dictionary, Awaitable, Constructor, StorageBackend, Cookie, QueueOperationInfo } from '@crawlee/types';
diff --git a/packages/core/src/memory-storage/memory-storage.ts b/packages/core/src/memory-storage/memory-storage.ts
index 4f39a0fb7..d321633ac 100644
--- a/packages/core/src/memory-storage/memory-storage.ts
+++ b/packages/core/src/memory-storage/memory-storage.ts
@@ -3,39 +3,39 @@ import { randomUUID } from 'node:crypto';
import type * as storage from '@crawlee/types';
import type { CrawleeLogger } from '@crawlee/types';
-import { DatasetClient } from './resource-clients/dataset.js';
-import { KeyValueStoreClient } from './resource-clients/key-value-store.js';
-import { RequestQueueClient } from './resource-clients/request-queue.js';
+import { DatasetBackend } from './resource-clients/dataset.js';
+import { KeyValueStoreBackend } from './resource-clients/key-value-store.js';
+import { RequestQueueBackend } from './resource-clients/request-queue.js';
export interface MemoryStorageOptions {
/**
- * Optional logger for MemoryStorageClient warnings.
+ * Optional logger for MemoryStorageBackend warnings.
*/
logger?: CrawleeLogger;
}
-export class MemoryStorageClient implements storage.StorageClient {
+export class MemoryStorageBackend implements storage.StorageBackend {
readonly logger?: CrawleeLogger;
/**
- * Unique per-instance cache partition key. Mirrors the way `FileSystemStorageClient` partitions its
- * cache by storage directory: two distinct `MemoryStorageClient` instances must not share cached clients.
+ * Unique per-instance cache partition key. Mirrors the way `FileSystemStorageBackend` partitions its
+ * cache by storage directory: two distinct `MemoryStorageBackend` instances must not share cached backends.
*/
- private readonly instanceCacheKey = `MemoryStorageClient:${randomUUID()}`;
+ private readonly instanceCacheKey = `MemoryStorageBackend:${randomUUID()}`;
- readonly keyValueStoreCache: KeyValueStoreClient[] = [];
- readonly datasetClientCache: DatasetClient[] = [];
- readonly requestQueueCache: RequestQueueClient[] = [];
+ readonly keyValueStoreBackendCache: KeyValueStoreBackend[] = [];
+ readonly datasetBackendCache: DatasetBackend[] = [];
+ readonly requestQueueBackendCache: RequestQueueBackend[] = [];
constructor(options: MemoryStorageOptions = {}) {
this.logger = options.logger;
}
/**
- * Return a per-instance unique cache key so that distinct `MemoryStorageClient` instances get separate
- * cache partitions in the storage-client cache.
+ * Return a per-instance unique cache key so that distinct `MemoryStorageBackend` instances get separate
+ * cache partitions in the storage backend cache.
*/
- getStorageClientCacheKey(): string {
+ getStorageBackendCacheKey(): string {
return this.instanceCacheKey;
}
@@ -50,11 +50,11 @@ export class MemoryStorageClient implements storage.StorageClient {
return { isAlias, cacheKey };
}
- async createDatasetClient(options: storage.CreateDatasetClientOptions = {}): Promise<storage.DatasetClient> {
- const { isAlias, cacheKey } = MemoryStorageClient.resolveStorageKey(options);
+ async createDatasetBackend(options: storage.CreateDatasetBackendOptions = {}): Promise<storage.DatasetBackend> {
+ const { isAlias, cacheKey } = MemoryStorageBackend.resolveStorageKey(options);
if (cacheKey) {
- const found = this.datasetClientCache.find(
+ const found = this.datasetBackendCache.find(
(store) =>
store.id === cacheKey ||
store.name?.toLowerCase() === cacheKey.toLowerCase() ||
@@ -65,23 +65,23 @@ export class MemoryStorageClient implements storage.StorageClient {
}
}
- const newStore = new DatasetClient({
+ const newStore = new DatasetBackend({
name: isAlias ? undefined : cacheKey,
cacheKey,
- client: this,
+ storageBackend: this,
});
- this.datasetClientCache.push(newStore);
+ this.datasetBackendCache.push(newStore);
return newStore;
}
- async createKeyValueStoreClient(
- options: storage.CreateKeyValueStoreClientOptions = {},
- ): Promise<storage.KeyValueStoreClient> {
- const { isAlias, cacheKey } = MemoryStorageClient.resolveStorageKey(options);
+ async createKeyValueStoreBackend(
+ options: storage.CreateKeyValueStoreBackendOptions = {},
+ ): Promise<storage.KeyValueStoreBackend> {
+ const { isAlias, cacheKey } = MemoryStorageBackend.resolveStorageKey(options);
if (cacheKey) {
- const found = this.keyValueStoreCache.find(
+ const found = this.keyValueStoreBackendCache.find(
(store) =>
store.id === cacheKey ||
store.name?.toLowerCase() === cacheKey.toLowerCase() ||
@@ -92,21 +92,23 @@ export class MemoryStorageClient implements storage.StorageClient {
}
}
- const newStore = new KeyValueStoreClient({
+ const newStore = new KeyValueStoreBackend({
name: isAlias ? undefined : cacheKey,
cacheKey,
- client: this,
+ storageBackend: this,
});
- this.keyValueStoreCache.push(newStore);
+ this.keyValueStoreBackendCache.push(newStore);
return newStore;
}
- async createRequestQueueClient(options: storage.CreateRequestQueueClientOptions = {}): Promise<RequestQueueClient> {
- const { isAlias, cacheKey } = MemoryStorageClient.resolveStorageKey(options);
+ async createRequestQueueBackend(
+ options: storage.CreateRequestQueueBackendOptions = {},
+ ): Promise<RequestQueueBackend> {
+ const { isAlias, cacheKey } = MemoryStorageBackend.resolveStorageKey(options);
if (cacheKey) {
- const found = this.requestQueueCache.find(
+ const found = this.requestQueueBackendCache.find(
(queue) =>
queue.id === cacheKey ||
queue.name?.toLowerCase() === cacheKey.toLowerCase() ||
@@ -117,42 +119,42 @@ export class MemoryStorageClient implements storage.StorageClient {
}
}
- const newStore = new RequestQueueClient({
+ const newStore = new RequestQueueBackend({
name: isAlias ? undefined : cacheKey,
cacheKey,
- client: this,
+ storageBackend: this,
});
- this.requestQueueCache.push(newStore);
+ this.requestQueueBackendCache.push(newStore);
return newStore;
}
async storageExists(id: string, type: 'Dataset' | 'KeyValueStore' | 'RequestQueue'): Promise<boolean> {
- let clients: { id: string }[];
+ let backends: { id: string }[];
switch (type) {
case 'Dataset':
- clients = this.datasetClientCache;
+ backends = this.datasetBackendCache;
break;
case 'KeyValueStore':
- clients = this.keyValueStoreCache;
+ backends = this.keyValueStoreBackendCache;
break;
case 'RequestQueue':
- clients = this.requestQueueCache;
+ backends = this.requestQueueBackendCache;
break;
default:
return false;
}
- // In-memory storage only knows about clients in its cache.
- return clients.some((store) => store.id === id);
+ // In-memory storage only knows about backends in its cache.
+ return backends.some((store) => store.id === id);
}
/**
* Cleans up the default storages before the run starts. For the in-memory storage this simply
* resets the in-memory state of the cached default dataset, key-value store and request queue.
*
- * As with `FileSystemStorageClient`, the run's input (the `INPUT` key in the default key-value
+ * As with `FileSystemStorageBackend`, the run's input (the `INPUT` key in the default key-value
* store) is preserved — only the rest of the default storages is cleared.
*/
async purge(): Promise<void> {
@@ -173,10 +175,10 @@ export class MemoryStorageClient implements storage.StorageClient {
await Promise.all([
// Preserve the run input (INPUT) when purging the default key-value store, matching
- // `FileSystemStorageClient`.
- purgeDefaults(this.keyValueStoreCache, async (store) => store.purgeExceptInput()),
- purgeDefaults(this.datasetClientCache, async (store) => store.purge()),
- purgeDefaults(this.requestQueueCache, async (store) => store.purge()),
+ // `FileSystemStorageBackend`.
+ purgeDefaults(this.keyValueStoreBackendCache, async (store) => store.purgeExceptInput()),
+ purgeDefaults(this.datasetBackendCache, async (store) => store.purge()),
+ purgeDefaults(this.requestQueueBackendCache, async (store) => store.purge()),
]);
}
diff --git a/packages/core/src/memory-storage/resource-clients/dataset.ts b/packages/core/src/memory-storage/resource-clients/dataset.ts
index 5bf2f7aca..14cd2684b 100644
--- a/packages/core/src/memory-storage/resource-clients/dataset.ts
+++ b/packages/core/src/memory-storage/resource-clients/dataset.ts
@@ -4,7 +4,7 @@ import type * as storage from '@crawlee/types';
import type { Dictionary } from '@crawlee/types';
import { s } from '@sapphire/shapeshift';
-import type { MemoryStorageClient } from '../memory-storage.js';
+import type { MemoryStorageBackend } from '../memory-storage.js';
import { BaseClient } from './common/base-client.js';
/**
@@ -19,7 +19,7 @@ const LIST_ITEMS_LIMIT = 999_999_999_999;
*/
const LOCAL_ENTRY_NAME_DIGITS = 9;
-export interface DatasetClientOptions {
+export interface DatasetBackendOptions {
id?: string;
name?: string;
/**
@@ -28,12 +28,12 @@ export interface DatasetClientOptions {
* metadata `name` (which is `undefined` for unnamed storages).
*/
cacheKey?: string;
- client: MemoryStorageClient;
+ storageBackend: MemoryStorageBackend;
}
-export class DatasetClient<Data extends Dictionary = Dictionary>
+export class DatasetBackend<Data extends Dictionary = Dictionary>
extends BaseClient
- implements storage.DatasetClient<Data>
+ implements storage.DatasetBackend<Data>
{
name?: string;
/**
@@ -47,13 +47,13 @@ export class DatasetClient<Data extends Dictionary = Dictionary>
itemCount = 0;
private readonly datasetEntries = new Map<string, Data>();
- private readonly client: MemoryStorageClient;
+ private readonly storageBackend: MemoryStorageBackend;
- constructor(options: DatasetClientOptions) {
+ constructor(options: DatasetBackendOptions) {
super(options.id ?? randomUUID());
this.name = options.name;
this.cacheKey = options.cacheKey ?? this.name ?? this.id;
- this.client = options.client;
+ this.storageBackend = options.storageBackend;
}
async getMetadata(): Promise<storage.DatasetInfo> {
@@ -62,12 +62,12 @@ export class DatasetClient<Data extends Dictionary = Dictionary>
}
async drop(): Promise<void> {
- const storeIndex = this.client.datasetClientCache.findIndex((store) => store.id === this.id);
+ const storeIndex = this.storageBackend.datasetBackendCache.findIndex((store) => store.id === this.id);
if (storeIndex !== -1) {
- const [oldClient] = this.client.datasetClientCache.splice(storeIndex, 1);
- oldClient.itemCount = 0;
- oldClient.datasetEntries.clear();
+ const [oldBackend] = this.storageBackend.datasetBackendCache.splice(storeIndex, 1);
+ oldBackend.itemCount = 0;
+ oldBackend.datasetEntries.clear();
}
}
@@ -78,7 +78,7 @@ export class DatasetClient<Data extends Dictionary = Dictionary>
this.updateTimestamps(true);
}
- getData(options: storage.DatasetClientListOptions = {}): Promise<storage.PaginatedList<Data>> {
+ getData(options: storage.DatasetBackendListOptions = {}): Promise<storage.PaginatedList<Data>> {
const { desc, limit, offset } = s
.object({
desc: s.boolean().optional(),
@@ -94,7 +94,7 @@ export class DatasetClient<Data extends Dictionary = Dictionary>
});
}
- private async getDataPage(options: storage.DatasetClientListOptions = {}): Promise<storage.PaginatedList<Data>> {
+ private async getDataPage(options: storage.DatasetBackendListOptions = {}): Promise<storage.PaginatedList<Data>> {
const { limit = LIST_ITEMS_LIMIT, offset = 0, desc } = options;
const [start, end] = this.getStartAndEndIndexes(
diff --git a/packages/core/src/memory-storage/resource-clients/key-value-store.ts b/packages/core/src/memory-storage/resource-clients/key-value-store.ts
index 80ea89fc0..87e608483 100644
--- a/packages/core/src/memory-storage/resource-clients/key-value-store.ts
+++ b/packages/core/src/memory-storage/resource-clients/key-value-store.ts
@@ -3,7 +3,7 @@ import { randomUUID } from 'node:crypto';
import type * as storage from '@crawlee/types';
import { s } from '@sapphire/shapeshift';
-import type { MemoryStorageClient } from '../memory-storage.js';
+import type { MemoryStorageBackend } from '../memory-storage.js';
import { isStream, toBuffer } from '../utils.js';
import { BaseClient } from './common/base-client.js';
import mime from 'mime-types';
@@ -12,11 +12,11 @@ const DEFAULT_LOCAL_FILE_EXTENSION = 'bin';
/**
* Key under which a run's input is stored in the default key-value store. Matches Crawlee's default
- * `inputKey` (`CRAWLEE_INPUT_KEY`) and the `INPUT` files `FileSystemStorageClient` preserves on purge.
+ * `inputKey` (`CRAWLEE_INPUT_KEY`) and the `INPUT` files `FileSystemStorageBackend` preserves on purge.
*/
const KEY_VALUE_STORE_INPUT_KEY = 'INPUT';
-export interface KeyValueStoreClientOptions {
+export interface KeyValueStoreBackendOptions {
name?: string;
id?: string;
/**
@@ -25,7 +25,7 @@ export interface KeyValueStoreClientOptions {
* metadata `name` (which is `undefined` for unnamed storages).
*/
cacheKey?: string;
- client: MemoryStorageClient;
+ storageBackend: MemoryStorageBackend;
}
export interface InternalKeyRecord {
@@ -35,7 +35,7 @@ export interface InternalKeyRecord {
extension: string;
}
-export class KeyValueStoreClient extends BaseClient implements storage.KeyValueStoreClient {
+export class KeyValueStoreBackend extends BaseClient implements storage.KeyValueStoreBackend {
name?: string;
/**
* The key used for cache lookup. For named storages, this equals the name. For alias (unnamed)
@@ -47,13 +47,13 @@ export class KeyValueStoreClient extends BaseClient implements storage.KeyValueS
modifiedAt = new Date();
private readonly keyValueEntries = new Map<string, InternalKeyRecord>();
- private readonly client: MemoryStorageClient;
+ private readonly storageBackend: MemoryStorageBackend;
- constructor(options: KeyValueStoreClientOptions) {
+ constructor(options: KeyValueStoreBackendOptions) {
super(options.id ?? randomUUID());
this.name = options.name;
this.cacheKey = options.cacheKey ?? this.name ?? this.id;
- this.client = options.client;
+ this.storageBackend = options.storageBackend;
}
async getMetadata(): Promise<storage.KeyValueStoreInfo> {
@@ -62,11 +62,11 @@ export class KeyValueStoreClient extends BaseClient implements storage.KeyValueS
}
async drop(): Promise<void> {
- const storeIndex = this.client.keyValueStoreCache.findIndex((store) => store.id === this.id);
+ const storeIndex = this.storageBackend.keyValueStoreBackendCache.findIndex((store) => store.id === this.id);
if (storeIndex !== -1) {
- const [oldClient] = this.client.keyValueStoreCache.splice(storeIndex, 1);
- oldClient.keyValueEntries.clear();
+ const [oldBackend] = this.storageBackend.keyValueStoreBackendCache.splice(storeIndex, 1);
+ oldBackend.keyValueEntries.clear();
}
}
@@ -76,8 +76,8 @@ export class KeyValueStoreClient extends BaseClient implements storage.KeyValueS
}
/**
- * Purges every record except the run's input. Used by {@link MemoryStorageClient.purge} for the
- * default key-value store, mirroring `FileSystemStorageClient`, which preserves `INPUT` (and its
+ * Purges every record except the run's input. Used by {@link MemoryStorageBackend.purge} for the
+ * default key-value store, mirroring `FileSystemStorageBackend`, which preserves `INPUT` (and its
* extension variants) when purging the default store. The in-memory key has no extension, so we
* preserve the bare `INPUT` key only.
*/
@@ -209,12 +209,12 @@ export class KeyValueStoreClient extends BaseClient implements storage.KeyValueS
const { key } = record;
let { value } = record;
// The frontend (KeyValueStore codec) serializes the value and resolves its content type
- // before it reaches the client. We only need it here for on-disk extension bookkeeping.
+ // before it reaches the backend. We only need it here for on-disk extension bookkeeping.
const contentType = record.contentType ?? 'application/octet-stream';
const extension = mime.extension(contentType) || DEFAULT_LOCAL_FILE_EXTENSION;
- // Draining a stream into a Buffer for storage is the client's responsibility.
+ // Draining a stream into a Buffer for storage is the backend's responsibility.
if (isStream(value)) {
const chunks = [];
for await (const chunk of value) {
@@ -223,7 +223,7 @@ export class KeyValueStoreClient extends BaseClient implements storage.KeyValueS
value = Buffer.concat(chunks);
}
- // This client is a byte transport: it stores and returns raw bytes regardless of the input
+ // This backend is a byte transport: it stores and returns raw bytes regardless of the input
// shape. Streams were drained above; encode strings to UTF-8 bytes and normalize
// ArrayBuffer / typed-array views to a Buffer over the same memory.
const normalizedValue: Buffer =
diff --git a/packages/core/src/memory-storage/resource-clients/request-queue.ts b/packages/core/src/memory-storage/resource-clients/request-queue.ts
index 81bf70ea0..9974bb779 100644
--- a/packages/core/src/memory-storage/resource-clients/request-queue.ts
+++ b/packages/core/src/memory-storage/resource-clients/request-queue.ts
@@ -3,7 +3,7 @@ import { randomUUID } from 'node:crypto';
import type * as storage from '@crawlee/types';
import { AsyncQueue } from '@sapphire/async-queue';
import { s } from '@sapphire/shapeshift';
-import type { MemoryStorageClient } from '../memory-storage.js';
+import type { MemoryStorageBackend } from '../memory-storage.js';
import { purgeNullsFromObject, uniqueKeyToRequestId } from '../utils.js';
import { BaseClient } from './common/base-client.js';
@@ -26,7 +26,7 @@ const requestOptionsShape = s.object({
forefront: s.boolean().optional(),
});
-export interface RequestQueueClientOptions {
+export interface RequestQueueBackendOptions {
name?: string;
id?: string;
/**
@@ -35,7 +35,7 @@ export interface RequestQueueClientOptions {
* metadata `name` (which is `undefined` for unnamed storages).
*/
cacheKey?: string;
- client: MemoryStorageClient;
+ storageBackend: MemoryStorageBackend;
}
export interface InternalRequest {
@@ -48,7 +48,7 @@ export interface InternalRequest {
json: string;
}
-export class RequestQueueClient extends BaseClient implements storage.RequestQueueClient {
+export class RequestQueueBackend extends BaseClient implements storage.RequestQueueBackend {
name?: string;
/**
* The key used for cache lookup. For named storages, this equals the name. For alias (unnamed)
@@ -61,7 +61,7 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQue
handledRequestCount = 0;
pendingRequestCount = 0;
/**
- * Serializes every operation that reads-then-writes this client's shared queue state — the
+ * Serializes every operation that reads-then-writes this backend's shared queue state — the
* `requests` map, the `forefrontRequestIds` array, the `inProgressRequestIds` set and the request
* counts. Those mutations span `await` points, so without this mutex a concurrent operation could
* interleave and corrupt them (e.g. a head scan pruning `forefrontRequestIds` while
@@ -82,13 +82,13 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQue
private readonly inProgressRequestIds = new Set<string>();
private readonly requests = new Map<string, InternalRequest>();
- private readonly client: MemoryStorageClient;
+ private readonly storageBackend: MemoryStorageBackend;
- constructor(options: RequestQueueClientOptions) {
+ constructor(options: RequestQueueBackendOptions) {
super(options.id ?? randomUUID());
this.name = options.name;
this.cacheKey = options.cacheKey ?? this.name ?? this.id;
- this.client = options.client;
+ this.storageBackend = options.storageBackend;
}
async getMetadata(): Promise<storage.RequestQueueInfo> {
@@ -103,17 +103,17 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQue
await this.queueStateMutex.wait();
try {
- const storeIndex = this.client.requestQueueCache.findIndex((queue) => queue.id === this.id);
+ const storeIndex = this.storageBackend.requestQueueBackendCache.findIndex((queue) => queue.id === this.id);
if (storeIndex !== -1) {
- const [oldClient] = this.client.requestQueueCache.splice(storeIndex, 1);
- oldClient.pendingRequestCount = 0;
+ const [oldBackend] = this.storageBackend.requestQueueBackendCache.splice(storeIndex, 1);
+ oldBackend.pendingRequestCount = 0;
// Clear all in-memory state, consistent with `purge`. Clearing `requests` alone would
// leave dangling ids in `forefrontRequestIds`/`inProgressRequestIds`, which a later head
// scan would resolve to a missing request and dereference.
- oldClient.requests.clear();
- oldClient.forefrontRequestIds = [];
- oldClient.inProgressRequestIds.clear();
+ oldBackend.requests.clear();
+ oldBackend.forefrontRequestIds = [];
+ oldBackend.inProgressRequestIds.clear();
}
} finally {
this.queueStateMutex.shift();
diff --git a/packages/core/src/service_locator.ts b/packages/core/src/service_locator.ts
index e2891fea1..79c3b1033 100644
--- a/packages/core/src/service_locator.ts
+++ b/packages/core/src/service_locator.ts
@@ -1,7 +1,7 @@
import { AsyncLocalStorage } from 'node:async_hooks';
-import { FileSystemStorageClient } from '@crawlee/fs-storage';
-import type { StorageClient } from '@crawlee/types';
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
+import type { StorageBackend } from '@crawlee/types';
import log from '@apify/log';
@@ -11,7 +11,7 @@ import type { EventManager } from './events/event_manager.js';
import { LocalEventManager } from './events/local_event_manager.js';
import type { CrawleeLogger } from './log.js';
import { ApifyLogAdapter } from './log.js';
-import { MemoryStorageClient } from './memory-storage/index.js';
+import { MemoryStorageBackend } from './memory-storage/index.js';
import { StorageInstanceManager } from './storages/storage_instance_manager.js';
interface ServiceLocatorInterface {
@@ -44,19 +44,19 @@ interface ServiceLocatorInterface {
setEventManager(eventManager: EventManager): void;
/**
- * Get the storage client.
- * Creates a default storage client if none has been set — `FileSystemStorageClient` when
- * `persistStorage` is enabled (the default), `MemoryStorageClient` otherwise.
+ * Get the storage backend.
+ * Creates a default storage backend if none has been set — `FileSystemStorageBackend` when
+ * `persistStorage` is enabled (the default), `MemoryStorageBackend` otherwise.
*/
- getStorageClient(): StorageClient;
+ getStorageBackend(): StorageBackend;
/**
- * Set the storage client.
+ * Set the storage backend.
*
- * @param storageClient The storage client to set
- * @throws {ServiceConflictError} If a different storage client has already been retrieved
+ * @param storageBackend The storage backend to set
+ * @throws {ServiceConflictError} If a different storage backend has already been retrieved
*/
- setStorageClient(storageClient: StorageClient): void;
+ setStorageBackend(storageBackend: StorageBackend): void;
/**
* Get the logger.
@@ -103,7 +103,7 @@ interface ServiceLocatorInterface {
* import { serviceLocator, BasicCrawler } from 'crawlee';
*
* // Optionally configure global services before creating crawlers
- * serviceLocator.setStorageClient(myCustomClient);
+ * serviceLocator.setStorageBackend(myCustomClient);
*
* // Crawler uses global services
* const crawler = new BasicCrawler({ ... });
@@ -111,12 +111,12 @@ interface ServiceLocatorInterface {
*
* **2. Per-crawler services (recommended for isolation):**
* ```typescript
- * import { BasicCrawler, Configuration, LocalEventManager, MemoryStorageClient } from 'crawlee';
+ * import { BasicCrawler, Configuration, LocalEventManager, MemoryStorageBackend } from 'crawlee';
*
* const crawler = new BasicCrawler({
* requestHandler: async ({ request }) => { ... },
* configuration: new Configuration({ ... }), // custom config
- * storageClient: new MemoryStorageClient(), // custom storage
+ * storageBackend: new MemoryStorageBackend(), // custom storage
* eventManager: LocalEventManager.fromConfig(), // custom events
* });
* // Crawler has its own isolated ServiceLocator instance
@@ -125,7 +125,7 @@ interface ServiceLocatorInterface {
export class ServiceLocator implements ServiceLocatorInterface {
private configuration?: Configuration;
private eventManager?: EventManager;
- private storageClient?: StorageClient;
+ private storageBackend?: StorageBackend;
private logger?: CrawleeLogger;
/**
@@ -140,18 +140,18 @@ export class ServiceLocator implements ServiceLocatorInterface {
*
* @param configuration Optional configuration instance to use
* @param eventManager Optional event manager instance to use
- * @param storageClient Optional storage client instance to use
+ * @param storageBackend Optional storage backend instance to use
* @param logger Optional logger instance to use
*/
constructor(
configuration?: Configuration,
eventManager?: EventManager,
- storageClient?: StorageClient,
+ storageBackend?: StorageBackend,
logger?: CrawleeLogger,
) {
this.configuration = configuration;
this.eventManager = eventManager;
- this.storageClient = storageClient;
+ this.storageBackend = storageBackend;
this.logger = logger;
}
@@ -205,43 +205,43 @@ export class ServiceLocator implements ServiceLocatorInterface {
this.eventManager = eventManager;
}
- getStorageClient(): StorageClient {
- if (!this.storageClient) {
+ getStorageBackend(): StorageBackend {
+ if (!this.storageBackend) {
this.getLogger().debug(
- 'No storage client set, implicitly creating and using the default storage client ' +
- '(FileSystemStorageClient when persistStorage is enabled, MemoryStorageClient otherwise).',
+ 'No storage backend set, implicitly creating and using the default storage backend ' +
+ '(FileSystemStorageBackend when persistStorage is enabled, MemoryStorageBackend otherwise).',
);
if (!this.configuration) {
this.getLogger().warning(
- 'Implicit creation of storage client will implicitly set configuration as side effect. ' +
+ 'Implicit creation of storage backend will implicitly set configuration as side effect. ' +
'It is advised to explicitly first set the configuration instead.',
);
}
const config = this.getConfiguration();
- this.storageClient = config.persistStorage
- ? new FileSystemStorageClient({
+ this.storageBackend = config.persistStorage
+ ? new FileSystemStorageBackend({
localDataDirectory: config.storageDir,
- logger: this.getLogger().child({ prefix: 'FileSystemStorageClient' }),
+ logger: this.getLogger().child({ prefix: 'FileSystemStorageBackend' }),
})
- : new MemoryStorageClient({
- logger: this.getLogger().child({ prefix: 'MemoryStorageClient' }),
+ : new MemoryStorageBackend({
+ logger: this.getLogger().child({ prefix: 'MemoryStorageBackend' }),
});
}
- return this.storageClient;
+ return this.storageBackend;
}
- setStorageClient(storageClient: StorageClient): void {
+ setStorageBackend(storageBackend: StorageBackend): void {
// Same instance, no need to do anything
- if (this.storageClient === storageClient) {
+ if (this.storageBackend === storageBackend) {
return;
}
- // Already have a different storage client that was retrieved
- if (this.storageClient) {
- throw new ServiceConflictError('StorageClient', storageClient, this.storageClient);
+ // Already have a different storage backend that was retrieved
+ if (this.storageBackend) {
+ throw new ServiceConflictError('StorageBackend', storageBackend, this.storageBackend);
}
- this.storageClient = storageClient;
+ this.storageBackend = storageBackend;
}
getLogger(): CrawleeLogger {
@@ -277,7 +277,7 @@ export class ServiceLocator implements ServiceLocatorInterface {
reset(): void {
this.configuration = undefined;
this.eventManager = undefined;
- this.storageClient = undefined;
+ this.storageBackend = undefined;
this.logger = undefined;
ServiceLocator.storageInstanceManager?.clearCache();
ServiceLocator.storageInstanceManager = undefined;
@@ -366,7 +366,7 @@ export const serviceLocator = new Proxy({} as ServiceLocatorInterface, {
},
set(_target, prop) {
throw new TypeError(
- `Cannot set property '${String(prop)}' on serviceLocator directly. Use the setter methods (e.g. setConfiguration(), setStorageClient()) instead.`,
+ `Cannot set property '${String(prop)}' on serviceLocator directly. Use the setter methods (e.g. setConfiguration(), setStorageBackend()) instead.`,
);
},
});
diff --git a/packages/core/src/storages/dataset.ts b/packages/core/src/storages/dataset.ts
index 1396cc162..174452bb3 100644
--- a/packages/core/src/storages/dataset.ts
+++ b/packages/core/src/storages/dataset.ts
@@ -1,4 +1,4 @@
-import type { DatasetClient, DatasetInfo, Dictionary, PaginatedList } from '@crawlee/types';
+import type { DatasetBackend, DatasetInfo, Dictionary, PaginatedList } from '@crawlee/types';
import { stringify } from 'csv-stringify/sync';
import ow from 'ow';
@@ -187,7 +187,7 @@ export interface DatasetExportToOptions extends DatasetExportOptions {
export class Dataset<Data extends Dictionary = Dictionary> {
id: string;
name?: string;
- client: DatasetClient<Data>;
+ backend: DatasetBackend<Data>;
log: CrawleeLogger;
private readonly statsTracker = new StorageStatsTracker<DatasetStats>({
@@ -204,13 +204,13 @@ export class Dataset<Data extends Dictionary = Dictionary> {
) {
this.id = options.id;
this.name = options.name;
- this.client = options.client;
+ this.backend = options.backend;
this.log = serviceLocator.getLogger().child({ prefix: 'Dataset' });
}
/**
* Backend-independent usage counters tracked for this dataset (read / write operations issued to
- * the underlying storage client). Counted per client call.
+ * the underlying storage backend). Counted per backend call.
*/
get stats(): DatasetStats {
return this.statsTracker.current;
@@ -239,7 +239,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {
}
this.statsTracker.add('writeCount');
- await this.client.pushData(items);
+ await this.backend.pushData(items);
}
/**
@@ -250,7 +250,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {
try {
this.statsTracker.add('readCount');
- return await this.client.getData(options);
+ return await this.backend.getData(options);
} catch (e) {
const error = e as Error;
if (error.message.includes('Cannot create a string longer than')) {
@@ -384,7 +384,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {
async getInfo(): Promise<DatasetInfo> {
checkStorageAccess();
- return this.client.getMetadata();
+ return this.backend.getMetadata();
}
/**
@@ -555,7 +555,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {
if (fetchLimit <= 0) break;
this.statsTracker.add('readCount');
- const page = await this.client.getData({ ...options, offset, limit: fetchLimit });
+ const page = await this.backend.getData({ ...options, offset, limit: fetchLimit });
yield page;
yielded += page.items.length;
@@ -651,7 +651,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {
async drop(): Promise<void> {
checkStorageAccess();
- await this.client.drop();
+ await this.backend.drop();
serviceLocator.getStorageInstanceManager().removeFromCache(this);
}
@@ -680,22 +680,22 @@ export class Dataset<Data extends Dictionary = Dictionary> {
options,
ow.object.exactShape({
config: ow.optional.object.instanceOf(Configuration),
- storageClient: ow.optional.object,
+ storageBackend: ow.optional.object,
}),
);
options.config ??= Configuration.getGlobalConfig();
- const client = options.storageClient ?? serviceLocator.getStorageClient();
+ const storageBackend = options.storageBackend ?? serviceLocator.getStorageBackend();
- await purgeDefaultStorages({ onlyPurgeOnce: true, client, config: options.config });
+ await purgeDefaultStorages({ onlyPurgeOnce: true, storageBackend, config: options.config });
- const resolved = await resolveStorageIdentifier(identifier, client, 'Dataset');
+ const resolved = await resolveStorageIdentifier(identifier, storageBackend, 'Dataset');
return serviceLocator.getStorageInstanceManager().openStorage<Dataset<Data>>(this, {
...resolved,
- clientOpener: () => client.createDatasetClient(resolved),
- clientCacheKey: client.getStorageClientCacheKey?.() ?? client.constructor.name,
+ backendOpener: () => storageBackend.createDatasetBackend(resolved),
+ backendCacheKey: storageBackend.getStorageBackendCacheKey?.() ?? storageBackend.constructor.name,
});
}
@@ -777,7 +777,7 @@ export interface DatasetReducer<T, Data> {
export interface DatasetOptions {
id: string;
name?: string;
- client: DatasetClient;
+ backend: DatasetBackend;
}
export interface DatasetContent<Data> {
diff --git a/packages/core/src/storages/key_value_store.ts b/packages/core/src/storages/key_value_store.ts
index 99f69bd06..b329c6101 100644
--- a/packages/core/src/storages/key_value_store.ts
+++ b/packages/core/src/storages/key_value_store.ts
@@ -1,4 +1,4 @@
-import type { Dictionary, KeyValueStoreClient, KeyValueStoreItemData } from '@crawlee/types';
+import type { Dictionary, KeyValueStoreBackend, KeyValueStoreItemData } from '@crawlee/types';
import ow, { ArgumentError } from 'ow';
import { KEY_VALUE_STORE_KEY_REGEX } from '@apify/consts';
@@ -78,7 +78,7 @@ const KVS_KEYS_DEFAULT_LIMIT = 1000;
export class KeyValueStore {
readonly id: string;
readonly name?: string;
- private readonly client: KeyValueStoreClient;
+ private readonly backend: KeyValueStoreBackend;
private persistStateEventStarted = false;
/** Cache for persistent (auto-saved) values. When we try to set such value, the cache will be updated automatically. */
@@ -100,12 +100,12 @@ export class KeyValueStore {
) {
this.id = options.id;
this.name = options.name;
- this.client = options.client;
+ this.backend = options.backend;
}
/**
* Backend-independent usage counters tracked for this key-value store (read / write / delete /
- * list operations issued to the underlying storage client). Counted per client call.
+ * list operations issued to the underlying storage backend). Counted per backend call.
*/
get stats(): KeyValueStoreStats {
return this.statsTracker.current;
@@ -212,7 +212,7 @@ export class KeyValueStore {
ow(key, ow.string.nonEmpty);
this.statsTracker.add('readCount');
- const record = await this.client.getValue(key);
+ const record = await this.backend.getValue(key);
// A missing record falls back to the default; a record that parses to a falsy value (including
// a stored literal `null`) is returned verbatim, so callers can tell "stored null" from "absent".
@@ -220,7 +220,7 @@ export class KeyValueStore {
return defaultValue ?? null;
}
- // Storage clients are byte transports — the value is raw bytes; the frontend parses it here.
+ // Storage backends are byte transports — the value is raw bytes; the frontend parses it here.
return parseValue(record.value, record.contentType ?? null) as T;
}
@@ -255,7 +255,7 @@ export class KeyValueStore {
ow(key, ow.string.nonEmpty);
this.statsTracker.add('readCount');
- const record = await this.client.getValue(key);
+ const record = await this.backend.getValue(key);
if (!record) return null;
return {
@@ -274,7 +274,7 @@ export class KeyValueStore {
checkStorageAccess();
ow(key, ow.string.nonEmpty);
- return this.client.recordExists(key);
+ return this.backend.recordExists(key);
}
async getAutoSavedValue<T extends Dictionary = Dictionary>(key: string, defaultValue = {} as T): Promise<T> {
@@ -330,7 +330,7 @@ export class KeyValueStore {
const results: T[] = [];
for (const item of page) {
this.statsTracker.add('readCount');
- const record = await this.client.getValue(item.key);
+ const record = await this.backend.getValue(item.key);
if (record) {
const parsed = parseValue(record.value, record.contentType ?? null);
results.push(mapRecord(item.key, parsed));
@@ -348,7 +348,7 @@ export class KeyValueStore {
while (true) {
this.statsTracker.add('listCount');
- const { items, isTruncated, nextExclusiveStartKey } = await this.client.listKeys({
+ const { items, isTruncated, nextExclusiveStartKey } = await this.backend.listKeys({
...options,
exclusiveStartKey,
limit,
@@ -449,13 +449,13 @@ export class KeyValueStore {
// In this case delete the record.
if (value === null) {
this.statsTracker.add('deleteCount');
- return this.client.deleteValue(key);
+ return this.backend.deleteValue(key);
}
const serialized = serializeValue(value, optionsCopy.contentType);
this.statsTracker.add('writeCount');
- return this.client.setValue({
+ return this.backend.setValue({
key,
value: serialized.value,
contentType: serialized.contentType,
@@ -469,7 +469,7 @@ export class KeyValueStore {
async drop(): Promise<void> {
checkStorageAccess();
- await this.client.drop();
+ await this.backend.drop();
serviceLocator.getStorageInstanceManager().removeFromCache(this);
}
@@ -643,7 +643,7 @@ export class KeyValueStore {
* @param key The key of the record to generate the public URL for.
*/
async getPublicUrl(key: string): Promise<string | undefined> {
- return this.client.getPublicUrl(key);
+ return this.backend.getPublicUrl(key);
}
/**
@@ -671,21 +671,21 @@ export class KeyValueStore {
options,
ow.object.exactShape({
config: ow.optional.object.instanceOf(Configuration),
- storageClient: ow.optional.object,
+ storageBackend: ow.optional.object,
}),
);
options.config ??= Configuration.getGlobalConfig();
- const client = options.storageClient ?? serviceLocator.getStorageClient();
+ const storageBackend = options.storageBackend ?? serviceLocator.getStorageBackend();
- await purgeDefaultStorages({ onlyPurgeOnce: true, client, config: options.config });
+ await purgeDefaultStorages({ onlyPurgeOnce: true, storageBackend, config: options.config });
- const resolved = await resolveStorageIdentifier(identifier, client, 'KeyValueStore');
+ const resolved = await resolveStorageIdentifier(identifier, storageBackend, 'KeyValueStore');
return serviceLocator.getStorageInstanceManager().openStorage<KeyValueStore>(this, {
...resolved,
- clientOpener: () => client.createKeyValueStoreClient(resolved),
- clientCacheKey: client.getStorageClientCacheKey?.() ?? client.constructor.name,
+ backendOpener: () => storageBackend.createKeyValueStoreBackend(resolved),
+ backendCacheKey: storageBackend.getStorageBackendCacheKey?.() ?? storageBackend.constructor.name,
});
}
@@ -891,7 +891,7 @@ export interface KeyConsumer {
export interface KeyValueStoreOptions {
id: string;
name?: string;
- client: KeyValueStoreClient;
+ backend: KeyValueStoreBackend;
}
/**
diff --git a/packages/core/src/storages/key_value_store_codec.ts b/packages/core/src/storages/key_value_store_codec.ts
index 07e6faec4..d17747af4 100644
--- a/packages/core/src/storages/key_value_store_codec.ts
+++ b/packages/core/src/storages/key_value_store_codec.ts
@@ -19,7 +19,7 @@ const STRINGIFIABLE_CONTENT_TYPE_RXS = [new RegExp(`^${CONTENT_TYPE_JSON}$`, 'i'
* - `string` → `text/plain; charset=utf-8` (passthrough)
* - anything else → `application/json; charset=utf-8` (serialized via `jsonStringifyExtended`)
*
- * Does NOT drain streams — that is storage mechanics and stays in the storage client.
+ * Does NOT drain streams — that is storage mechanics and stays in the storage backend.
*
* Backend-independent.
*/
diff --git a/packages/core/src/storages/request_manager.ts b/packages/core/src/storages/request_manager.ts
index ca858db73..601d84c7b 100644
--- a/packages/core/src/storages/request_manager.ts
+++ b/packages/core/src/storages/request_manager.ts
@@ -36,7 +36,7 @@ export interface IRequestManager extends IRequestLoader {
* Tells the manager how long a consumer expects to hold a request fetched via `fetchNextRequest()`
* before marking it handled or reclaiming it (typically the request-handler timeout plus padding).
*
- * Managers backed by a storage client that reserves requests via locking use this to avoid handing
+ * Managers backed by a storage backend that reserves requests via locking use this to avoid handing
* the same request out again while it is still being processed. Implementations that do not need
* this hint may leave it `undefined`.
*/
diff --git a/packages/core/src/storages/request_queue.ts b/packages/core/src/storages/request_queue.ts
index 461249eb7..a7995dc3a 100644
--- a/packages/core/src/storages/request_queue.ts
+++ b/packages/core/src/storages/request_queue.ts
@@ -6,7 +6,7 @@ import type {
Dictionary,
ProcessedRequest,
QueueOperationInfo,
- RequestQueueClient,
+ RequestQueueBackend,
RequestQueueInfo,
} from '@crawlee/types';
import {
@@ -43,7 +43,7 @@ import { resolveStorageIdentifier } from './storage_instance_manager.js';
import { getRequestId, purgeDefaultStorages } from './utils.js';
/**
- * The maximum number of requests cached locally to avoid redundant calls to the storage client.
+ * The maximum number of requests cached locally to avoid redundant calls to the storage backend.
* @internal
*/
const MAX_CACHED_REQUESTS = 2_000_000;
@@ -93,7 +93,7 @@ export class RequestQueue implements IStorage, IRequestManager {
name?: string;
timeoutSecs = 30;
clientKey = cryptoRandomObjectId();
- client: RequestQueueClient;
+ backend: RequestQueueBackend;
protected proxyConfiguration?: ProxyConfiguration;
log: CrawleeLogger;
@@ -109,7 +109,7 @@ export class RequestQueue implements IStorage, IRequestManager {
/**
* The largest expected request-processing time (in seconds) seen so far via
* {@link setExpectedRequestProcessingTimeSecs}. Used to ensure that value is only ever raised, never
- * lowered, before being forwarded to the storage client.
+ * lowered, before being forwarded to the storage backend.
*/
protected expectedRequestProcessingSecs = 0;
@@ -124,7 +124,7 @@ export class RequestQueue implements IStorage, IRequestManager {
/**
* Backend-independent usage counters tracked for this request queue (write operations and
- * queue-head reads issued to the underlying storage client). Counted per client call.
+ * queue-head reads issued to the underlying storage backend). Counted per backend call.
*/
get stats(): RequestQueueStats {
return this.statsTracker.current;
@@ -140,7 +140,7 @@ export class RequestQueue implements IStorage, IRequestManager {
this.id = options.id;
this.name = options.name;
this.events = serviceLocator.getEventManager();
- this.client = options.client;
+ this.backend = options.backend;
this.proxyConfiguration = options.proxyConfiguration;
@@ -236,7 +236,7 @@ export class RequestQueue implements IStorage, IRequestManager {
}
this.statsTracker.add('writeCount');
- const { processedRequests } = await this.client.addBatchOfRequests([request], { forefront });
+ const { processedRequests } = await this.backend.addBatchOfRequests([request], { forefront });
const queueOperationInfo = {
...processedRequests[0],
uniqueKey: request.uniqueKey,
@@ -343,7 +343,7 @@ export class RequestQueue implements IStorage, IRequestManager {
}
this.statsTracker.add('writeCount');
- const apiResults = await this.client.addBatchOfRequests([...requestsToAdd.values()], { forefront });
+ const apiResults = await this.backend.addBatchOfRequests([...requestsToAdd.values()], { forefront });
// Report unprocessed requests
results.unprocessedRequests = apiResults.unprocessedRequests;
@@ -577,7 +577,7 @@ export class RequestQueue implements IStorage, IRequestManager {
ow(uniqueKey, ow.string);
- const requestOptions = await this.client.getRequest(uniqueKey);
+ const requestOptions = await this.backend.getRequest(uniqueKey);
if (!requestOptions) return null;
return new Request(requestOptions as unknown as RequestOptions);
@@ -608,7 +608,7 @@ export class RequestQueue implements IStorage, IRequestManager {
}
this.statsTracker.add('headItemReadCount');
- const requestOptions = await this.client.fetchNextRequest();
+ const requestOptions = await this.backend.fetchNextRequest();
if (!requestOptions) return null;
return new Request(requestOptions as unknown as RequestOptions);
@@ -636,7 +636,7 @@ export class RequestQueue implements IStorage, IRequestManager {
const handledAt = request.handledAt ?? new Date().toISOString();
this.statsTracker.add('writeCount');
- const processedRequest = await this.client.markRequestAsHandled({
+ const processedRequest = await this.backend.markRequestAsHandled({
...request,
handledAt,
});
@@ -688,7 +688,7 @@ export class RequestQueue implements IStorage, IRequestManager {
const { forefront = false } = options;
this.statsTracker.add('writeCount');
- const processedRequest = await this.client.reclaimRequest(request, { forefront });
+ const processedRequest = await this.backend.reclaimRequest(request, { forefront });
// The request was not in progress — nothing to reclaim.
if (!processedRequest) {
@@ -718,7 +718,7 @@ export class RequestQueue implements IStorage, IRequestManager {
async isEmpty(): Promise<boolean> {
checkStorageAccess();
- return this.client.isEmpty();
+ return this.backend.isEmpty();
}
/**
@@ -737,12 +737,12 @@ export class RequestQueue implements IStorage, IRequestManager {
return false;
}
- return this.client.isFinished();
+ return this.backend.isFinished();
}
/**
* Tells the queue how long a consumer expects to hold a fetched request before marking it handled
- * or reclaiming it (typically the request-handler timeout plus padding), so that a storage client
+ * or reclaiming it (typically the request-handler timeout plus padding), so that a storage backend
* that reserves requests via locking does not hand the same request out again while it is still
* being processed.
*
@@ -756,7 +756,7 @@ export class RequestQueue implements IStorage, IRequestManager {
}
this.expectedRequestProcessingSecs = secs;
- await this.client.setExpectedRequestProcessingTimeSecs?.(secs);
+ await this.backend.setExpectedRequestProcessingTimeSecs?.(secs);
}
/**
@@ -783,7 +783,7 @@ export class RequestQueue implements IStorage, IRequestManager {
async drop(): Promise<void> {
checkStorageAccess();
- await this.client.drop();
+ await this.backend.drop();
serviceLocator.getStorageInstanceManager().removeFromCache(this);
}
@@ -794,7 +794,7 @@ export class RequestQueue implements IStorage, IRequestManager {
async purge(): Promise<void> {
checkStorageAccess();
- await this.client.purge();
+ await this.backend.purge();
// Reset in-memory bookkeeping so the queue behaves as if freshly opened.
this.requestCache.clear();
@@ -855,7 +855,7 @@ export class RequestQueue implements IStorage, IRequestManager {
async getInfo(): Promise<RequestQueueInfo> {
checkStorageAccess();
- return this.client.getMetadata();
+ return this.backend.getMetadata();
}
/**
@@ -949,36 +949,36 @@ export class RequestQueue implements IStorage, IRequestManager {
options,
ow.object.exactShape({
config: ow.optional.object.instanceOf(Configuration),
- storageClient: ow.optional.object,
+ storageBackend: ow.optional.object,
proxyConfiguration: ow.optional.object,
httpClient: ow.optional.object,
}),
);
- const client = options.storageClient ?? serviceLocator.getStorageClient();
+ const storageBackend = options.storageBackend ?? serviceLocator.getStorageBackend();
const config = options.config ?? serviceLocator.getConfiguration();
- await purgeDefaultStorages({ onlyPurgeOnce: true, client, config });
+ await purgeDefaultStorages({ onlyPurgeOnce: true, storageBackend, config });
- const resolved = await resolveStorageIdentifier(identifier, client, 'RequestQueue');
+ const resolved = await resolveStorageIdentifier(identifier, storageBackend, 'RequestQueue');
const queue = await serviceLocator
.getStorageInstanceManager()
.openStorage<RequestQueue>(this as unknown as Constructor<RequestQueue>, {
...resolved,
- clientOpener: () => client.createRequestQueueClient(resolved),
- clientCacheKey: client.getStorageClientCacheKey?.() ?? client.constructor.name,
+ backendOpener: () => storageBackend.createRequestQueueBackend(resolved),
+ backendCacheKey: storageBackend.getStorageBackendCacheKey?.() ?? storageBackend.constructor.name,
});
queue.proxyConfiguration = options.proxyConfiguration;
queue.httpClient = options.httpClient;
if (!queue.isInitialized) {
- // Re-create the request queue client with clientKey and timeoutSecs so that
+ // Re-create the request queue backend with clientKey and timeoutSecs so that
// request locking works correctly for API-backed implementations.
// TODO: clientKey/timeoutSecs are Apify-platform concerns and should eventually be pushed
// down into the Apify SDK's client implementation, aligning with crawlee-python's approach
- // where locking is handled internally by the client (see crawlee-python PR #1194).
- queue.client = await client.createRequestQueueClient({
+ // where locking is handled internally by the backend (see crawlee-python PR #1194).
+ queue.backend = await storageBackend.createRequestQueueBackend({
id: queue.id,
clientKey: queue.clientKey,
timeoutSecs: queue.timeoutSecs,
@@ -1003,7 +1003,7 @@ interface RequestLruItem {
export interface RequestQueueOptions {
id: string;
name?: string;
- client: RequestQueueClient;
+ backend: RequestQueueBackend;
/**
* Used to pass the proxy configuration for the `requestsFromUrl` objects.
diff --git a/packages/core/src/storages/storage_instance_manager.ts b/packages/core/src/storages/storage_instance_manager.ts
index 7bf85b13c..ee4a14c8c 100644
--- a/packages/core/src/storages/storage_instance_manager.ts
+++ b/packages/core/src/storages/storage_instance_manager.ts
@@ -1,8 +1,8 @@
import type {
- DatasetClient,
- KeyValueStoreClient,
- RequestQueueClient,
- StorageClient,
+ DatasetBackend,
+ KeyValueStoreBackend,
+ RequestQueueBackend,
+ StorageBackend,
StorageIdentifier,
} from '@crawlee/types';
import { AsyncQueue } from '@sapphire/async-queue';
@@ -29,7 +29,7 @@ type CacheTier = Map<Constructor<IStorage>, Map<string, Map<Hashable, IStorage>>
/**
* Three-tier cache for storage instances, modelled after crawlee-python's `_StorageCache`.
*
- * Each tier maps `[storageClass][key][clientCacheKey] → instance`:
+ * Each tier maps `[storageClass][key][backendCacheKey] → instance`:
* - `byId` — keyed by the backend-assigned storage id
* - `byName` — keyed by the persistent storage name
* - `byAlias` — keyed by a run-scoped alias (e.g. `'__default__'` for unnamed storages)
@@ -45,12 +45,12 @@ class StorageCache {
id,
name,
alias,
- clientCacheKey,
+ backendCacheKey,
}: (
| { id: string; name?: string; alias?: undefined }
| { id?: string; name: string; alias?: undefined }
| { id?: undefined; name?: undefined; alias: string }
- ) & { clientCacheKey: Hashable },
+ ) & { backendCacheKey: Hashable },
): T | undefined {
for (const [tier, key] of [
[this.byId, id],
@@ -58,7 +58,7 @@ class StorageCache {
[this.byAlias, alias],
] as [CacheTier, string | undefined][]) {
if (key === undefined) continue;
- const cached = tier.get(cls)?.get(key)?.get(clientCacheKey);
+ const cached = tier.get(cls)?.get(key)?.get(backendCacheKey);
if (cached) {
if (cached instanceof (cls as unknown as abstract new (...args: any[]) => any)) {
return cached as T;
@@ -76,29 +76,29 @@ class StorageCache {
cls: Constructor<T>,
key: string,
instance: T,
- clientCacheKey: Hashable,
+ backendCacheKey: Hashable,
): void {
if (!tier.has(cls)) tier.set(cls, new Map());
const keyMap = tier.get(cls)!;
if (!keyMap.has(key)) keyMap.set(key, new Map());
- keyMap.get(key)!.set(clientCacheKey, instance);
+ keyMap.get(key)!.set(backendCacheKey, instance);
}
/**
* Cache an instance under its actual id, name, and an optional alias.
*/
- set<T extends IStorage>(cls: Constructor<T>, instance: T, clientCacheKey: Hashable, alias?: string): void {
+ set<T extends IStorage>(cls: Constructor<T>, instance: T, backendCacheKey: Hashable, alias?: string): void {
// Always cache by id.
- this.setInMap(this.byId, cls, instance.id, instance, clientCacheKey);
+ this.setInMap(this.byId, cls, instance.id, instance, backendCacheKey);
// Cache by name — only for named storages.
if (instance.name) {
- this.setInMap(this.byName, cls, instance.name, instance, clientCacheKey);
+ this.setInMap(this.byName, cls, instance.name, instance, backendCacheKey);
}
// Cache by alias — only for unnamed storages opened via alias.
if (alias !== undefined) {
- this.setInMap(this.byAlias, cls, alias, instance, clientCacheKey);
+ this.setInMap(this.byAlias, cls, alias, instance, backendCacheKey);
}
}
@@ -125,10 +125,10 @@ class StorageCache {
*/
checkNameAliasConflict<T extends IStorage>(
cls: Constructor<T>,
- { name, alias, clientCacheKey }: { name?: string; alias?: string; clientCacheKey: Hashable },
+ { name, alias, backendCacheKey }: { name?: string; alias?: string; backendCacheKey: Hashable },
): void {
if (alias) {
- const existingByName = this.byName.get(cls)?.get(alias)?.get(clientCacheKey);
+ const existingByName = this.byName.get(cls)?.get(alias)?.get(backendCacheKey);
if (existingByName) {
throw new Error(
`Cannot open storage with alias "${alias}" because a named storage with the same identifier already exists.`,
@@ -136,7 +136,7 @@ class StorageCache {
}
}
if (name) {
- const existingByAlias = this.byAlias.get(cls)?.get(name)?.get(clientCacheKey);
+ const existingByAlias = this.byAlias.get(cls)?.get(name)?.get(backendCacheKey);
if (existingByAlias) {
throw new Error(
`Cannot open storage with name "${name}" because an alias storage with the same identifier already exists.` +
@@ -172,10 +172,10 @@ class StorageCache {
* Unified manager for opening and caching storage instances (Dataset, KeyValueStore, RequestQueue).
*
* A single instance manages all storage types. Instances are cached by
- * `(storageClass, id/name/alias, clientCacheKey)` so the same storage is never opened twice.
+ * `(storageClass, id/name/alias, backendCacheKey)` so the same storage is never opened twice.
*
* The manager itself does not resolve identifiers — callers pass explicit `id`, `name`, or `alias` (at most one),
- * and a pre-bound `clientOpener` promise. When none of `id`, `name`, `alias` are provided, the manager automatically
+ * and a pre-bound `backendOpener` promise. When none of `id`, `name`, `alias` are provided, the manager automatically
* assigns a reserved default alias.
*
* @ignore
@@ -192,9 +192,9 @@ export class StorageInstanceManager {
* @param name Storage name (mutually exclusive with `id` and `alias`).
* @param alias Run-scoped alias (mutually exclusive with `id` and `name`).
* Automatically assigned when no identifier is provided.
- * @param clientOpener A **lazy** factory that creates the sub-client.
+ * @param backendOpener A **lazy** factory that creates the sub-backend.
* Only called on a cache miss.
- * @param clientCacheKey Opaque key identifying the storage backend, so that the same logical
+ * @param backendCacheKey Opaque key identifying the storage backend, so that the same logical
* storage opened through different clients is cached separately.
*/
async openStorage<TStorage extends IStorage>(
@@ -203,11 +203,11 @@ export class StorageInstanceManager {
id,
name,
alias,
- clientOpener,
- clientCacheKey,
+ backendOpener,
+ backendCacheKey,
}: (ExplicitStorageIdentifier | DefaultStorageIdentifier) & {
- clientOpener: () => Promise<DatasetClient | KeyValueStoreClient | RequestQueueClient>;
- clientCacheKey: Hashable;
+ backendOpener: () => Promise<DatasetBackend | KeyValueStoreBackend | RequestQueueBackend>;
+ backendCacheKey: Hashable;
},
): Promise<TStorage> {
// Auto-set alias='__default__' when no parameters are specified (mirrors crawlee-python).
@@ -217,18 +217,18 @@ export class StorageInstanceManager {
// Fast-path cache check (no lock).
if (alias !== undefined) {
- const cached = this.cache.get(cls, { alias, clientCacheKey });
+ const cached = this.cache.get(cls, { alias, backendCacheKey });
if (cached) return cached;
} else if (id) {
- const cached = this.cache.get(cls, { id, clientCacheKey });
+ const cached = this.cache.get(cls, { id, backendCacheKey });
if (cached) return cached;
} else if (name) {
- const cached = this.cache.get(cls, { name, clientCacheKey });
+ const cached = this.cache.get(cls, { name, backendCacheKey });
if (cached) return cached;
}
const identifierKey = id ?? name ?? alias ?? DEFAULT_STORAGE_ALIAS;
- const lockKey = `${cls.name}:${identifierKey}:${clientCacheKey}`;
+ const lockKey = `${cls.name}:${identifierKey}:${backendCacheKey}`;
if (!this.openerLocks.has(lockKey)) {
this.openerLocks.set(lockKey, new AsyncQueue());
@@ -239,33 +239,33 @@ export class StorageInstanceManager {
try {
// Double-check cache under lock (another caller may have filled it while we waited).
if (alias !== undefined) {
- const cached = this.cache.get(cls, { alias, clientCacheKey });
+ const cached = this.cache.get(cls, { alias, backendCacheKey });
if (cached) return cached;
} else if (id) {
- const cached = this.cache.get(cls, { id, clientCacheKey });
+ const cached = this.cache.get(cls, { id, backendCacheKey });
if (cached) return cached;
} else if (name) {
- const cached = this.cache.get(cls, { name, clientCacheKey });
+ const cached = this.cache.get(cls, { name, backendCacheKey });
if (cached) return cached;
}
// Prevent the same string from being used as both a name and an alias.
- this.cache.checkNameAliasConflict(cls, { name, alias, clientCacheKey });
+ this.cache.checkNameAliasConflict(cls, { name, alias, backendCacheKey });
- // Cache miss — create the sub-client and storage instance.
- const subClient = await clientOpener();
+ // Cache miss — create the sub-backend and storage instance.
+ const subBackend = await backendOpener();
const storageInfo = await (
- subClient as DatasetClient | KeyValueStoreClient | RequestQueueClient
+ subBackend as DatasetBackend | KeyValueStoreBackend | RequestQueueBackend
).getMetadata();
const instance = new cls({
id: storageInfo.id,
name: storageInfo.name,
- client: subClient,
+ backend: subBackend,
}) as TStorage;
// Atomic cache writes (no awaits between these).
- this.cache.set(cls, instance, clientCacheKey, alias);
+ this.cache.set(cls, instance, backendCacheKey, alias);
return instance;
} finally {
@@ -323,7 +323,7 @@ export interface DefaultStorageIdentifier {
/**
* Decompose a user-provided `identifier` (the `Dataset.open()` / `KeyValueStore.open()` /
* `RequestQueue.open()` argument) into separate `id`, `name`, and `alias` fields that
- * the `StorageInstanceManager` and `StorageClient.create*Client` expect.
+ * the `StorageInstanceManager` and `StorageBackend.create*Client` expect.
*
* - `null` / `undefined` / `{}` → default storage alias
* - `string` → resolved via `storageExists` (ID-first, then name)
@@ -333,7 +333,7 @@ export interface DefaultStorageIdentifier {
*/
export async function resolveStorageIdentifier(
identifier: string | StorageIdentifier | null | undefined,
- client: StorageClient,
+ storageBackend: StorageBackend,
storageType: 'Dataset' | 'KeyValueStore' | 'RequestQueue',
): Promise<ExplicitStorageIdentifier> {
if (identifier === null || identifier === undefined) {
@@ -341,7 +341,7 @@ export async function resolveStorageIdentifier(
}
if (typeof identifier === 'string') {
- if (client.storageExists && (await client.storageExists(identifier, storageType))) {
+ if (storageBackend.storageExists && (await storageBackend.storageExists(identifier, storageType))) {
return { id: identifier };
}
return { name: identifier };
diff --git a/packages/core/src/storages/storage_stats.ts b/packages/core/src/storages/storage_stats.ts
index 1ecc29753..cbf1ff686 100644
--- a/packages/core/src/storages/storage_stats.ts
+++ b/packages/core/src/storages/storage_stats.ts
@@ -2,7 +2,7 @@
* Backend-independent usage counters tracked by the storage frontend classes
* ({@apilink Dataset}, {@apilink KeyValueStore}, {@apilink RequestQueue}).
*
- * These count the operations the frontend issues against its underlying storage client, so they are
+ * These count the operations the frontend issues against its underlying storage backend, so they are
* meaningful for any storage backend (memory, file system, cloud). They are tallied per client call
* — e.g. iterating a key-value store increments `readCount` once per record fetched and `listCount`
* once per listed page. Backend-specific figures that the frontend cannot compute (such as the number
diff --git a/packages/core/src/storages/utils.ts b/packages/core/src/storages/utils.ts
index bb7631076..83c5e7dfa 100644
--- a/packages/core/src/storages/utils.ts
+++ b/packages/core/src/storages/utils.ts
@@ -1,6 +1,6 @@
import crypto from 'node:crypto';
-import type { BaseHttpClient, Dictionary, StorageClient } from '@crawlee/types';
+import type { BaseHttpClient, Dictionary, StorageBackend } from '@crawlee/types';
import { Configuration } from '../configuration.js';
import type { ProxyConfiguration } from '../proxy_configuration.js';
@@ -16,7 +16,7 @@ interface PurgeDefaultStorageOptions {
*/
onlyPurgeOnce?: boolean;
config?: Configuration;
- client?: StorageClient;
+ storageBackend?: StorageBackend;
}
/**
@@ -27,7 +27,7 @@ interface PurgeDefaultStorageOptions {
* explicitly, e.g. via `RequestList.open()`). We can disable that via `purgeOnStart` {@apilink Configuration}
* option or by setting `CRAWLEE_PURGE_ON_START` environment variable to `0` or `false`.
*
- * This is a shortcut for running (optional) `purge` method on the StorageClient interface, in other words
+ * This is a shortcut for running (optional) `purge` method on the StorageBackend interface, in other words
* it will call the `purge` method of the underlying storage implementation we are currently using. You can
* make sure the storage is purged only once for a given execution context if you set `onlyPurgeOnce` to `true` in
* the `options` object
@@ -41,25 +41,25 @@ export async function purgeDefaultStorages(options?: PurgeDefaultStorageOptions)
* explicitly, e.g. via `RequestList.open()`). We can disable that via `purgeOnStart` {@apilink Configuration}
* option or by setting `CRAWLEE_PURGE_ON_START` environment variable to `0` or `false`.
*
- * This is a shortcut for running (optional) `purge` method on the StorageClient interface, in other words
+ * This is a shortcut for running (optional) `purge` method on the StorageBackend interface, in other words
* it will call the `purge` method of the underlying storage implementation we are currently using.
*/
-export async function purgeDefaultStorages(config?: Configuration, client?: StorageClient): Promise<void>;
+export async function purgeDefaultStorages(config?: Configuration, storageBackend?: StorageBackend): Promise<void>;
export async function purgeDefaultStorages(
configOrOptions?: Configuration | PurgeDefaultStorageOptions,
- client?: StorageClient,
+ storageBackend?: StorageBackend,
) {
const options: PurgeDefaultStorageOptions =
configOrOptions instanceof Configuration
? {
- client,
+ storageBackend,
config: configOrOptions,
}
: (configOrOptions ?? {});
const { config = serviceLocator.getConfiguration(), onlyPurgeOnce = false } = options;
- ({ client = serviceLocator.getStorageClient() } = options);
+ ({ storageBackend = serviceLocator.getStorageBackend() } = options);
- const casted = client as StorageClient & { __purged?: Promise<void> };
+ const casted = storageBackend as StorageBackend & { __purged?: Promise<void> };
const runPurge = async () => {
try {
@@ -232,9 +232,9 @@ export interface StorageOpenOptions {
config?: Configuration;
/**
- * Optional storage client that should be used to open storages.
+ * Optional storage backend that should be used to open storages.
*/
- storageClient?: StorageClient;
+ storageBackend?: StorageBackend;
/**
* Used to pass the proxy configuration for the `requestsFromUrl` objects.
diff --git a/packages/core/test/core/service_locator.test.ts b/packages/core/test/core/service_locator.test.ts
index 2b3436e79..905d90b91 100644
--- a/packages/core/test/core/service_locator.test.ts
+++ b/packages/core/test/core/service_locator.test.ts
@@ -3,12 +3,12 @@ import {
ApifyLogAdapter,
Configuration,
LocalEventManager,
- MemoryStorageClient,
+ MemoryStorageBackend,
ServiceConflictError,
ServiceLocator,
serviceLocator,
} from '@crawlee/core';
-import { FileSystemStorageClient } from '@crawlee/fs-storage';
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
function makeMockLogger(overrides: Partial<CrawleeLogger> = {}): CrawleeLogger {
const logger: CrawleeLogger = {
@@ -130,43 +130,43 @@ describe('ServiceLocator', () => {
});
});
- describe('StorageClient', () => {
- test('default storage client', () => {
- const defaultStorageClient = serviceLocator.getStorageClient();
- expect(defaultStorageClient).toBeInstanceOf(FileSystemStorageClient);
+ describe('StorageBackend', () => {
+ test('default storage backend', () => {
+ const defaultStorageBackend = serviceLocator.getStorageBackend();
+ expect(defaultStorageBackend).toBeInstanceOf(FileSystemStorageBackend);
});
- test('custom storage client', () => {
- const customStorageClient = new MemoryStorageClient();
- serviceLocator.setStorageClient(customStorageClient);
- const storageClient = serviceLocator.getStorageClient();
+ test('custom storage backend', () => {
+ const customStorageBackend = new MemoryStorageBackend();
+ serviceLocator.setStorageBackend(customStorageBackend);
+ const storageBackend = serviceLocator.getStorageBackend();
- expect(storageClient).toBe(customStorageClient);
+ expect(storageBackend).toBe(customStorageBackend);
});
- test('storage client overwrite not possible', () => {
- const customStorageClient = new MemoryStorageClient();
- serviceLocator.setStorageClient(customStorageClient);
+ test('storage backend overwrite not possible', () => {
+ const customStorageBackend = new MemoryStorageBackend();
+ serviceLocator.setStorageBackend(customStorageBackend);
- const anotherCustomStorageClient = new MemoryStorageClient();
+ const anotherCustomStorageBackend = new MemoryStorageBackend();
expect(() => {
- serviceLocator.setStorageClient(anotherCustomStorageClient);
+ serviceLocator.setStorageBackend(anotherCustomStorageBackend);
}).toThrow(ServiceConflictError);
});
- test('storage client conflict', () => {
- // Retrieve storage client first
- serviceLocator.getStorageClient();
+ test('storage backend conflict', () => {
+ // Retrieve storage backend first
+ serviceLocator.getStorageBackend();
- const customStorageClient = new MemoryStorageClient();
+ const customStorageBackend = new MemoryStorageBackend();
expect(() => {
- serviceLocator.setStorageClient(customStorageClient);
+ serviceLocator.setStorageBackend(customStorageBackend);
}).toThrow(ServiceConflictError);
expect(() => {
- serviceLocator.setStorageClient(customStorageClient);
- }).toThrow(/StorageClient is already in use/);
+ serviceLocator.setStorageBackend(customStorageBackend);
+ }).toThrow(/StorageBackend is already in use/);
});
});
@@ -206,9 +206,9 @@ describe('ServiceLocator', () => {
}).toThrow(/Logger is already in use/);
});
- test('setting logger after getStorageClient throws ServiceConflictError (logger already locked)', () => {
- // getStorageClient() implicitly calls getLogger(), locking the logger
- serviceLocator.getStorageClient();
+ test('setting logger after getStorageBackend throws ServiceConflictError (logger already locked)', () => {
+ // getStorageBackend() implicitly calls getLogger(), locking the logger
+ serviceLocator.getStorageBackend();
const customLogger = makeMockLogger();
@@ -239,16 +239,16 @@ describe('ServiceLocator', () => {
persistStateIntervalMillis: 1000,
systemInfoIntervalMillis: 1000,
});
- const customStorageClient = new MemoryStorageClient();
+ const customStorageBackend = new MemoryStorageBackend();
serviceLocator.setConfiguration(customConfig);
serviceLocator.setEventManager(customEventManager);
- serviceLocator.setStorageClient(customStorageClient);
+ serviceLocator.setStorageBackend(customStorageBackend);
// Verify they're set
expect(serviceLocator.getConfiguration()).toBe(customConfig);
expect(serviceLocator.getEventManager()).toBe(customEventManager);
- expect(serviceLocator.getStorageClient()).toBe(customStorageClient);
+ expect(serviceLocator.getStorageBackend()).toBe(customStorageBackend);
expect(serviceLocator.getLogger()).toBe(customLogger);
// Reset
@@ -287,14 +287,14 @@ describe('ServiceLocator', () => {
}).not.toThrow();
});
- test('setting same storage client instance is allowed', () => {
- const storageClient = new MemoryStorageClient();
- serviceLocator.setStorageClient(storageClient);
- serviceLocator.getStorageClient();
+ test('setting same storage backend instance is allowed', () => {
+ const storageBackend = new MemoryStorageBackend();
+ serviceLocator.setStorageBackend(storageBackend);
+ serviceLocator.getStorageBackend();
// Setting the same instance again should not throw
expect(() => {
- serviceLocator.setStorageClient(storageClient);
+ serviceLocator.setStorageBackend(storageBackend);
}).not.toThrow();
});
@@ -343,7 +343,7 @@ describe('ServiceLocator', () => {
describe('Per-crawler ServiceLocator', () => {
test('creating separate service locator for crawler', () => {
const crawlerConfig = new Configuration({ headless: false });
- const crawlerStorage = new MemoryStorageClient();
+ const crawlerStorage = new MemoryStorageBackend();
const crawlerEvents = new LocalEventManager({
persistStateIntervalMillis: 1000,
systemInfoIntervalMillis: 1000,
@@ -353,7 +353,7 @@ describe('ServiceLocator', () => {
expect(crawlerLocator.getConfiguration()).toBe(crawlerConfig);
expect(crawlerLocator.getEventManager()).toBe(crawlerEvents);
- expect(crawlerLocator.getStorageClient()).toBe(crawlerStorage);
+ expect(crawlerLocator.getStorageBackend()).toBe(crawlerStorage);
// Global service locator should remain independent
expect(serviceLocator.getConfiguration()).not.toBe(crawlerConfig);
diff --git a/packages/core/test/enqueue_links/add-enqueue-strategy-to-requests.ts b/packages/core/test/enqueue_links/add-enqueue-strategy-to-requests.ts
index bea08e92c..eaeda0eb7 100644
--- a/packages/core/test/enqueue_links/add-enqueue-strategy-to-requests.ts
+++ b/packages/core/test/enqueue_links/add-enqueue-strategy-to-requests.ts
@@ -4,7 +4,7 @@ import { cheerioCrawlerEnqueueLinks, EnqueueStrategy, RequestQueue, serviceLocat
import log from '@apify/log';
-const apifyClient = serviceLocator.getStorageClient();
+const apifyClient = serviceLocator.getStorageBackend();
const HTML = `
<html>
@@ -25,7 +25,7 @@ const HTML = `
function createRequestQueueMock() {
const enqueued: Source[] = [];
- const requestQueue = new RequestQueue({ id: 'xxx', client: apifyClient });
+ const requestQueue = new RequestQueue({ id: 'xxx', backend: apifyClient });
// @ts-expect-error Override method for testing
requestQueue.addRequests = async function (requests) {
diff --git a/packages/core/test/enqueue_links/protocol-matching-based-on-strategy.test.ts b/packages/core/test/enqueue_links/protocol-matching-based-on-strategy.test.ts
index 70c379d95..3908e062b 100644
--- a/packages/core/test/enqueue_links/protocol-matching-based-on-strategy.test.ts
+++ b/packages/core/test/enqueue_links/protocol-matching-based-on-strategy.test.ts
@@ -4,7 +4,7 @@ import { cheerioCrawlerEnqueueLinks, EnqueueStrategy, RequestQueue, serviceLocat
import log from '@apify/log';
-const apifyClient = serviceLocator.getStorageClient();
+const apifyClient = serviceLocator.getStorageBackend();
const HTML = `
<html>
@@ -22,7 +22,7 @@ const HTML = `
function createRequestQueueMock() {
const enqueued: Source[] = [];
- const requestQueue = new RequestQueue({ id: 'xxx', client: apifyClient }, serviceLocator.getConfiguration());
+ const requestQueue = new RequestQueue({ id: 'xxx', backend: apifyClient }, serviceLocator.getConfiguration());
// @ts-expect-error Override method for testing
requestQueue.addRequests = async function (requests) {
diff --git a/packages/core/test/enqueue_links/user-provided-patterns-with-enqueue-strategy.test.ts b/packages/core/test/enqueue_links/user-provided-patterns-with-enqueue-strategy.test.ts
index 468fedffe..877f0f186 100644
--- a/packages/core/test/enqueue_links/user-provided-patterns-with-enqueue-strategy.test.ts
+++ b/packages/core/test/enqueue_links/user-provided-patterns-with-enqueue-strategy.test.ts
@@ -4,7 +4,7 @@ import { cheerioCrawlerEnqueueLinks, EnqueueStrategy, RequestQueue, serviceLocat
import log from '@apify/log';
-const apifyClient = serviceLocator.getStorageClient();
+const apifyClient = serviceLocator.getStorageBackend();
const HTML = `
<html>
@@ -34,7 +34,7 @@ const HTML = `
function createRequestQueueMock() {
const enqueued: Source[] = [];
- const requestQueue = new RequestQueue({ id: 'xxx', client: apifyClient }, serviceLocator.getConfiguration());
+ const requestQueue = new RequestQueue({ id: 'xxx', backend: apifyClient }, serviceLocator.getConfiguration());
// @ts-expect-error Override method for testing
requestQueue.addRequests = async function (requests) {
diff --git a/packages/core/test/enqueue_links/userData.test.ts b/packages/core/test/enqueue_links/userData.test.ts
index 282d3910a..0ef548816 100644
--- a/packages/core/test/enqueue_links/userData.test.ts
+++ b/packages/core/test/enqueue_links/userData.test.ts
@@ -5,7 +5,7 @@ import { load } from 'cheerio';
import log from '@apify/log';
-const apifyClient = serviceLocator.getStorageClient();
+const apifyClient = serviceLocator.getStorageBackend();
const HTML = `
<html>
@@ -23,7 +23,7 @@ const HTML = `
function createRequestQueueMock() {
const enqueued: Source[] = [];
- const requestQueue = new RequestQueue({ id: 'xxx', client: apifyClient }, serviceLocator.getConfiguration());
+ const requestQueue = new RequestQueue({ id: 'xxx', backend: apifyClient }, serviceLocator.getConfiguration());
// @ts-expect-error Override method for testing
requestQueue.addRequests = async function (requests) {
diff --git a/packages/core/test/memory-storage/async-iteration.test.ts b/packages/core/test/memory-storage/async-iteration.test.ts
index ca012ffca..fc5239f16 100644
--- a/packages/core/test/memory-storage/async-iteration.test.ts
+++ b/packages/core/test/memory-storage/async-iteration.test.ts
@@ -1,15 +1,15 @@
-import { MemoryStorageClient } from '@crawlee/core';
-import type { DatasetClient, KeyValueStoreClient } from '@crawlee/types';
+import { MemoryStorageBackend } from '@crawlee/core';
+import type { DatasetBackend, KeyValueStoreBackend } from '@crawlee/types';
describe('Async iteration support', () => {
- const storage = new MemoryStorageClient();
+ const storage = new MemoryStorageBackend();
describe('Dataset.getData', () => {
const elements = Array.from({ length: 25 }, (_, i) => ({ index: i }));
- let dataset: DatasetClient<{ index: number }>;
+ let dataset: DatasetBackend<{ index: number }>;
beforeAll(async () => {
- dataset = (await storage.createDatasetClient({ name: 'async-iteration-dataset' })) as DatasetClient<{
+ dataset = (await storage.createDatasetBackend({ name: 'async-iteration-dataset' })) as DatasetBackend<{
index: number;
}>;
await dataset.pushData(elements);
@@ -55,10 +55,10 @@ describe('Async iteration support', () => {
describe('KeyValueStore.listKeys', () => {
const keys = Array.from({ length: 25 }, (_, i) => `key-${String(i).padStart(2, '0')}`);
- let kvStore: KeyValueStoreClient;
+ let kvStore: KeyValueStoreBackend;
beforeAll(async () => {
- kvStore = await storage.createKeyValueStoreClient({ name: 'async-iteration-kvs' });
+ kvStore = await storage.createKeyValueStoreBackend({ name: 'async-iteration-kvs' });
for (const key of keys) {
// The client is a byte-transport: pass serialized bytes + content type.
diff --git a/packages/core/test/memory-storage/key-value-store/purge.test.ts b/packages/core/test/memory-storage/key-value-store/purge.test.ts
index c9cfe0fb0..9fa058fcc 100644
--- a/packages/core/test/memory-storage/key-value-store/purge.test.ts
+++ b/packages/core/test/memory-storage/key-value-store/purge.test.ts
@@ -1,10 +1,10 @@
-import { MemoryStorageClient } from '@crawlee/core';
-import type { KeyValueStoreClient } from '@crawlee/types';
+import { MemoryStorageBackend } from '@crawlee/core';
+import type { KeyValueStoreBackend } from '@crawlee/types';
-describe('MemoryStorageClient.purge preserves the default key-value store input', () => {
+describe('MemoryStorageBackend.purge preserves the default key-value store input', () => {
test('purging keeps INPUT in the default store but removes everything else', async () => {
- const storage = new MemoryStorageClient();
- const store: KeyValueStoreClient = await storage.createKeyValueStoreClient({ name: 'default' });
+ const storage = new MemoryStorageBackend();
+ const store: KeyValueStoreBackend = await storage.createKeyValueStoreBackend({ name: 'default' });
await store.setValue({
key: 'INPUT',
@@ -19,7 +19,7 @@ describe('MemoryStorageClient.purge preserves the default key-value store input'
await storage.purge();
- // INPUT must survive the purge (parity with FileSystemStorageClient)...
+ // INPUT must survive the purge (parity with FileSystemStorageBackend)...
const input = await store.getValue('INPUT');
expect(input?.value.toString()).toBe(JSON.stringify({ hello: 'world' }));
@@ -30,8 +30,8 @@ describe('MemoryStorageClient.purge preserves the default key-value store input'
});
test('purging a non-default store removes INPUT as well', async () => {
- const storage = new MemoryStorageClient();
- const store: KeyValueStoreClient = await storage.createKeyValueStoreClient({ name: 'not-default' });
+ const storage = new MemoryStorageBackend();
+ const store: KeyValueStoreBackend = await storage.createKeyValueStoreBackend({ name: 'not-default' });
await store.setValue({
key: 'INPUT',
@@ -39,7 +39,7 @@ describe('MemoryStorageClient.purge preserves the default key-value store input'
contentType: 'application/json; charset=utf-8',
});
- // `purge` on the storage client only touches default storages, so a named store keeps its data.
+ // `purge` on the storage backend only touches default storages, so a named store keeps its data.
await storage.purge();
expect((await store.getValue('INPUT'))?.value.toString()).toBe(JSON.stringify({ hello: 'world' }));
diff --git a/packages/core/test/memory-storage/key-value-store/stream.test.ts b/packages/core/test/memory-storage/key-value-store/stream.test.ts
index cd9f9bb41..5c361dd2e 100644
--- a/packages/core/test/memory-storage/key-value-store/stream.test.ts
+++ b/packages/core/test/memory-storage/key-value-store/stream.test.ts
@@ -1,14 +1,14 @@
import { Readable } from 'node:stream';
-import { MemoryStorageClient } from '@crawlee/core';
+import { MemoryStorageBackend } from '@crawlee/core';
describe('KeyValueStore should drain streams when setting records', () => {
- const storage = new MemoryStorageClient();
+ const storage = new MemoryStorageBackend();
const fsStream = Readable.from([Buffer.from('hello'), Buffer.from('world')]);
test('should drain stream', async () => {
- const defaultStore = await storage.createKeyValueStoreClient({ name: 'default' });
+ const defaultStore = await storage.createKeyValueStoreBackend({ name: 'default' });
await defaultStore.setValue({ key: 'streamz', value: fsStream, contentType: 'text/plain' });
diff --git a/packages/core/test/memory-storage/no-crash-on-big-buffers.test.ts b/packages/core/test/memory-storage/no-crash-on-big-buffers.test.ts
index df7b9790c..8997f698c 100644
--- a/packages/core/test/memory-storage/no-crash-on-big-buffers.test.ts
+++ b/packages/core/test/memory-storage/no-crash-on-big-buffers.test.ts
@@ -1,16 +1,16 @@
// https://github.com/apify/crawlee/issues/1732
// https://github.com/apify/crawlee/issues/1710
-import { MemoryStorageClient } from '@crawlee/core';
-import type { KeyValueStoreClient } from '@crawlee/types';
+import { MemoryStorageBackend } from '@crawlee/core';
+import type { KeyValueStoreBackend } from '@crawlee/types';
-describe('MemoryStorageClient should not crash when saving a big buffer', () => {
- const storage = new MemoryStorageClient();
+describe('MemoryStorageBackend should not crash when saving a big buffer', () => {
+ const storage = new MemoryStorageBackend();
- let store: KeyValueStoreClient;
+ let store: KeyValueStoreBackend;
beforeAll(async () => {
- store = await storage.createKeyValueStoreClient();
+ store = await storage.createKeyValueStoreBackend();
});
test('should not crash when saving a big buffer', async () => {
diff --git a/packages/core/test/memory-storage/request-queue/forefront.test.ts b/packages/core/test/memory-storage/request-queue/forefront.test.ts
index 7ab2f861b..69be37769 100644
--- a/packages/core/test/memory-storage/request-queue/forefront.test.ts
+++ b/packages/core/test/memory-storage/request-queue/forefront.test.ts
@@ -2,14 +2,14 @@ import { rm } from 'node:fs/promises';
import { resolve } from 'node:path';
import { setTimeout as sleep } from 'node:timers/promises';
-import { MemoryStorageClient } from '@crawlee/core';
-import type { RequestQueueClient } from '@crawlee/types';
+import { MemoryStorageBackend } from '@crawlee/core';
+import type { RequestQueueBackend } from '@crawlee/types';
/**
* Drains the queue via `fetchNextRequest`, marking each request as handled, and returns the
* pathnames in the order they were served.
*/
-async function fetchOrder(client: RequestQueueClient): Promise<string[]> {
+async function fetchOrder(client: RequestQueueBackend): Promise<string[]> {
const order: string[] = [];
for (let request = await client.fetchNextRequest(); request != null; request = await client.fetchNextRequest()) {
@@ -21,12 +21,12 @@ async function fetchOrder(client: RequestQueueClient): Promise<string[]> {
}
describe('RequestQueue respects `forefront` when fetching requests', () => {
- const storage = new MemoryStorageClient();
+ const storage = new MemoryStorageBackend();
- let requestQueue: RequestQueueClient;
+ let requestQueue: RequestQueueBackend;
beforeEach(async () => {
- requestQueue = await storage.createRequestQueueClient({ name: 'forefront' });
+ requestQueue = await storage.createRequestQueueBackend({ name: 'forefront' });
});
afterEach(async () => {
@@ -146,12 +146,12 @@ describe('RequestQueue respects `forefront` when fetching requests', () => {
});
describe('RequestQueue holds fetched requests in progress', () => {
- const storage = new MemoryStorageClient();
+ const storage = new MemoryStorageBackend();
- let requestQueue: RequestQueueClient;
+ let requestQueue: RequestQueueBackend;
beforeEach(async () => {
- requestQueue = await storage.createRequestQueueClient({ name: 'in-progress' });
+ requestQueue = await storage.createRequestQueueBackend({ name: 'in-progress' });
});
afterEach(async () => {
diff --git a/packages/core/test/memory-storage/request-queue/handledRequestCount-should-update.test.ts b/packages/core/test/memory-storage/request-queue/handledRequestCount-should-update.test.ts
index 5992ae895..47cd4710a 100644
--- a/packages/core/test/memory-storage/request-queue/handledRequestCount-should-update.test.ts
+++ b/packages/core/test/memory-storage/request-queue/handledRequestCount-should-update.test.ts
@@ -1,13 +1,13 @@
-import { MemoryStorageClient } from '@crawlee/core';
-import type { RequestQueueClient } from '@crawlee/types';
+import { MemoryStorageBackend } from '@crawlee/core';
+import type { RequestQueueBackend } from '@crawlee/types';
describe('RequestQueue handledRequestCount should update', () => {
- const storage = new MemoryStorageClient();
+ const storage = new MemoryStorageBackend();
- let requestQueue: RequestQueueClient;
+ let requestQueue: RequestQueueBackend;
beforeAll(async () => {
- requestQueue = await storage.createRequestQueueClient({ name: 'handledRequestCount' });
+ requestQueue = await storage.createRequestQueueBackend({ name: 'handledRequestCount' });
});
test('after marking a request as handled, it should increment the handledRequestCount', async () => {
diff --git a/packages/core/test/memory-storage/request-queue/in-progress.test.ts b/packages/core/test/memory-storage/request-queue/in-progress.test.ts
index 9d2eb5cd9..935359823 100644
--- a/packages/core/test/memory-storage/request-queue/in-progress.test.ts
+++ b/packages/core/test/memory-storage/request-queue/in-progress.test.ts
@@ -1,10 +1,10 @@
-import { MemoryStorageClient } from '@crawlee/core';
-import type { RequestQueueClient } from '@crawlee/types';
+import { MemoryStorageBackend } from '@crawlee/core';
+import type { RequestQueueBackend } from '@crawlee/types';
describe('RequestQueue in-progress requests', () => {
test('a fetched request stays in progress until it is handled or reclaimed', async () => {
- const storage = new MemoryStorageClient();
- const queue: RequestQueueClient = await storage.createRequestQueueClient({ name: 'in-progress' });
+ const storage = new MemoryStorageBackend();
+ const queue: RequestQueueBackend = await storage.createRequestQueueBackend({ name: 'in-progress' });
await queue.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }]);
@@ -21,8 +21,8 @@ describe('RequestQueue in-progress requests', () => {
});
test('a fetched request becomes fetchable again once reclaimed', async () => {
- const storage = new MemoryStorageClient();
- const queue: RequestQueueClient = await storage.createRequestQueueClient({ name: 'reclaim' });
+ const storage = new MemoryStorageBackend();
+ const queue: RequestQueueBackend = await storage.createRequestQueueBackend({ name: 'reclaim' });
await queue.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }]);
@@ -38,8 +38,8 @@ describe('RequestQueue in-progress requests', () => {
});
test('an in-progress request can be marked as handled', async () => {
- const storage = new MemoryStorageClient();
- const queue: RequestQueueClient = await storage.createRequestQueueClient({ name: 'handle' });
+ const storage = new MemoryStorageBackend();
+ const queue: RequestQueueBackend = await storage.createRequestQueueBackend({ name: 'handle' });
await queue.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }]);
@@ -59,8 +59,8 @@ describe('RequestQueue in-progress requests', () => {
});
test('multiple requests are each handed out only once while in progress', async () => {
- const storage = new MemoryStorageClient();
- const queue: RequestQueueClient = await storage.createRequestQueueClient({ name: 'multi' });
+ const storage = new MemoryStorageBackend();
+ const queue: RequestQueueBackend = await storage.createRequestQueueBackend({ name: 'multi' });
await queue.addBatchOfRequests([
{ url: 'http://example.com/1', uniqueKey: '1' },
@@ -81,8 +81,8 @@ describe('RequestQueue in-progress requests', () => {
});
test('dropping a queue with a pending forefront request does not corrupt later head scans', async () => {
- const storage = new MemoryStorageClient();
- const queue: RequestQueueClient = await storage.createRequestQueueClient({ name: 'drop-forefront' });
+ const storage = new MemoryStorageBackend();
+ const queue: RequestQueueBackend = await storage.createRequestQueueBackend({ name: 'drop-forefront' });
// A forefront request leaves an id in `forefrontRequestIds`. `drop` must clear that alongside the
// `requests` map, otherwise a later head scan would resolve the dangling id to a missing request
diff --git a/packages/core/test/memory-storage/reverse-datataset-list.test.ts b/packages/core/test/memory-storage/reverse-datataset-list.test.ts
index ed51e4bba..2ea7006fa 100644
--- a/packages/core/test/memory-storage/reverse-datataset-list.test.ts
+++ b/packages/core/test/memory-storage/reverse-datataset-list.test.ts
@@ -1,15 +1,15 @@
-import { MemoryStorageClient } from '@crawlee/core';
-import type { DatasetClient } from '@crawlee/types';
+import { MemoryStorageBackend } from '@crawlee/core';
+import type { DatasetBackend } from '@crawlee/types';
const elements = Array.from({ length: 10 }, (_, i) => ({ number: i }));
describe('Dataset#getData respects the desc option', () => {
- const storage = new MemoryStorageClient();
+ const storage = new MemoryStorageBackend();
- let dataset: DatasetClient;
+ let dataset: DatasetBackend;
beforeAll(async () => {
- dataset = await storage.createDatasetClient({ name: 'false' });
+ dataset = await storage.createDatasetBackend({ name: 'false' });
await dataset.pushData(elements);
});
diff --git a/packages/core/test/request-queue/adding-the-same-request-should-not-call-the-api.test.ts b/packages/core/test/request-queue/adding-the-same-request-should-not-call-the-api.test.ts
index 6e5ba6ad8..c2092ddea 100644
--- a/packages/core/test/request-queue/adding-the-same-request-should-not-call-the-api.test.ts
+++ b/packages/core/test/request-queue/adding-the-same-request-should-not-call-the-api.test.ts
@@ -1,22 +1,22 @@
-import { MemoryStorageClient } from '@crawlee/core';
-import type { RequestQueueClient } from '@crawlee/types';
+import { MemoryStorageBackend } from '@crawlee/core';
+import type { RequestQueueBackend } from '@crawlee/types';
import { RequestQueue, serviceLocator } from 'crawlee';
-let rqClient: RequestQueueClient;
+let rqClient: RequestQueueBackend;
beforeEach(async () => {
- const storage = new MemoryStorageClient();
- serviceLocator.setStorageClient(storage);
- rqClient = await storage.createRequestQueueClient({ name: 'test-request-queue-not-called-on-cached-request' });
+ const storage = new MemoryStorageBackend();
+ serviceLocator.setStorageBackend(storage);
+ rqClient = await storage.createRequestQueueBackend({ name: 'test-request-queue-not-called-on-cached-request' });
});
describe('RequestQueue#addRequest should not call the API if the request is already in the queue', () => {
test('should not call the API if the request is already in the queue', async () => {
const config = serviceLocator.getConfiguration();
const rqInfo = await rqClient.getMetadata();
- const requestQueue = new RequestQueue({ id: rqInfo.id, client: rqClient }, config);
+ const requestQueue = new RequestQueue({ id: rqInfo.id, backend: rqClient }, config);
- const clientSpy = vitest.spyOn(requestQueue.client, 'addBatchOfRequests');
+ const clientSpy = vitest.spyOn(requestQueue.backend, 'addBatchOfRequests');
await requestQueue.addRequest({ url: 'https://example.com' });
@@ -37,9 +37,9 @@ describe('RequestQueue#addRequests should not call the API if the request is alr
test('should not call the API if the request is already in the queue', async () => {
const config = serviceLocator.getConfiguration();
const rqInfo = await rqClient.getMetadata();
- const requestQueue = new RequestQueue({ id: rqInfo.id, client: rqClient }, config);
+ const requestQueue = new RequestQueue({ id: rqInfo.id, backend: rqClient }, config);
- const clientSpy = vitest.spyOn(requestQueue.client, 'addBatchOfRequests');
+ const clientSpy = vitest.spyOn(requestQueue.backend, 'addBatchOfRequests');
await requestQueue.addRequests([{ url: 'https://example2.com' }]);
diff --git a/packages/core/test/request-queue/request-queue.test.ts b/packages/core/test/request-queue/request-queue.test.ts
index 92db62db6..9a761b5bd 100644
--- a/packages/core/test/request-queue/request-queue.test.ts
+++ b/packages/core/test/request-queue/request-queue.test.ts
@@ -1,14 +1,14 @@
-import { MemoryStorageClient } from '@crawlee/core';
+import { MemoryStorageBackend } from '@crawlee/core';
import { RequestQueue } from 'crawlee';
import type { MockInstance } from 'vitest';
-const storage = new MemoryStorageClient();
+const storage = new MemoryStorageBackend();
async function makeQueue(name: string, numOfRequestsToAdd = 0) {
- const rqClient = await storage.createRequestQueueClient({ name });
+ const rqClient = await storage.createRequestQueueBackend({ name });
const rqInfo = await rqClient.getMetadata();
- const queue = new RequestQueue({ id: rqInfo.id, client: rqClient });
+ const queue = new RequestQueue({ id: rqInfo.id, backend: rqClient });
if (numOfRequestsToAdd) {
await queue.addRequests(
@@ -23,11 +23,11 @@ vitest.setConfig({ restoreMocks: false });
describe('RequestQueue#fetchNextRequest delegates to the client', () => {
let queue: RequestQueue;
- let clientFetchNextSpy: MockInstance<typeof queue.client.fetchNextRequest>;
+ let clientFetchNextSpy: MockInstance<typeof queue.backend.fetchNextRequest>;
beforeAll(async () => {
queue = await makeQueue('fetch-next-request', 1);
- clientFetchNextSpy = vitest.spyOn(queue.client, 'fetchNextRequest');
+ clientFetchNextSpy = vitest.spyOn(queue.backend, 'fetchNextRequest');
});
test('returns the first request via the client', async () => {
diff --git a/packages/core/test/storages/open-storage-with-different-client-should-be-respected.test.ts b/packages/core/test/storages/open-storage-with-different-client-should-be-respected.test.ts
index 24aab26a6..04c1769e5 100644
--- a/packages/core/test/storages/open-storage-with-different-client-should-be-respected.test.ts
+++ b/packages/core/test/storages/open-storage-with-different-client-should-be-respected.test.ts
@@ -1,29 +1,29 @@
-import { MemoryStorageClient } from '@crawlee/core';
+import { MemoryStorageBackend } from '@crawlee/core';
import { RequestQueue, serviceLocator } from 'crawlee';
-let newClient: MemoryStorageClient;
+let newClient: MemoryStorageBackend;
beforeEach(() => {
- newClient = new MemoryStorageClient();
- serviceLocator.setStorageClient(newClient);
+ newClient = new MemoryStorageBackend();
+ serviceLocator.setStorageBackend(newClient);
});
-describe('Opening a storage with a different storage client should be respected', () => {
+describe('Opening a storage with a different storage backend should be respected', () => {
test('opening a RequestQueue with default client from Configuration', async () => {
const queue = await RequestQueue.open({ name: 'test-rq-open-client-from-config' });
- // The sub-client should have been created by newClient (MemoryStorageClient),
- // so its internal `client` field should reference newClient.
- expect((queue.client as any).client).toBe(newClient);
+ // The sub-backend should have been created by newClient (MemoryStorageBackend),
+ // so its internal `storageBackend` field should reference newClient.
+ expect((queue.backend as any).storageBackend).toBe(newClient);
});
test('opening a RequestQueue with a different client', async () => {
- const thirdClient = new MemoryStorageClient();
+ const thirdClient = new MemoryStorageBackend();
// @ts-expect-error Using this to ensure the test/impl works
thirdClient._name = 'third-client';
- const queue = await RequestQueue.open({ name: 'test-rq-open-custom-client' }, { storageClient: thirdClient });
+ const queue = await RequestQueue.open({ name: 'test-rq-open-custom-client' }, { storageBackend: thirdClient });
- expect((queue.client as any).client).toBe(thirdClient);
+ expect((queue.backend as any).storageBackend).toBe(thirdClient);
});
});
diff --git a/packages/fs-storage/src/file-system-storage.ts b/packages/fs-storage/src/file-system-storage.ts
index 88fadd719..71561e6c8 100644
--- a/packages/fs-storage/src/file-system-storage.ts
+++ b/packages/fs-storage/src/file-system-storage.ts
@@ -6,13 +6,13 @@ import type { CrawleeLogger } from '@crawlee/types';
import { s } from '@sapphire/shapeshift';
import {
- FileSystemDatasetClient as NativeDatasetClient,
- FileSystemKeyValueStoreClient as NativeKeyValueStoreClient,
- FileSystemRequestQueueClient as NativeRequestQueueClient,
+ FileSystemDatasetClient as NativeDatasetBackend,
+ FileSystemKeyValueStoreClient as NativeKeyValueStoreBackend,
+ FileSystemRequestQueueClient as NativeRequestQueueBackend,
} from '@crawlee/fs-storage-native';
-import { DatasetClient } from './resource-clients/dataset.js';
-import { KeyValueStoreClient } from './resource-clients/key-value-store.js';
-import { RequestQueueClient } from './resource-clients/request-queue.js';
+import { DatasetBackend } from './resource-clients/dataset.js';
+import { KeyValueStoreBackend } from './resource-clients/key-value-store.js';
+import { RequestQueueBackend } from './resource-clients/request-queue.js';
export interface FileSystemStorageOptions {
/**
@@ -21,12 +21,12 @@ export interface FileSystemStorageOptions {
localDataDirectory: string;
/**
- * Optional logger for FileSystemStorageClient warnings.
+ * Optional logger for FileSystemStorageBackend warnings.
*/
logger?: CrawleeLogger;
/**
- * How the on-disk request queues opened by this client are expected to be accessed.
+ * How the on-disk request queues opened by this backend are expected to be accessed.
*
* With `'single'` (the default), this process asserts it is the *sole* consumer of every request
* queue it opens: on open, any requests that a previous run left *in progress* (e.g. after a
@@ -44,14 +44,14 @@ export interface FileSystemStorageOptions {
}
/**
- * A file-system storage client backed by the native `@crawlee/fs-storage-native` Rust extension.
+ * A file-system storage backend backed by the native `@crawlee/fs-storage-native` Rust extension.
*
* The native extension owns the on-disk format, timestamps, item counting, request-queue locking and
* state persistence. This class is responsible for resolving the user-facing `id` / `name` / `alias`
- * identifiers to native storages, caching the opened clients (so that `storageExists`, `purge` and
+ * identifiers to native storages, caching the opened backends (so that `storageExists`, `purge` and
* `teardown` can operate over them), and exposing them through the `@crawlee/types` interfaces.
*/
-export class FileSystemStorageClient implements storage.StorageClient {
+export class FileSystemStorageBackend implements storage.StorageBackend {
readonly localDataDirectory: string;
readonly datasetsDirectory: string;
readonly keyValueStoresDirectory: string;
@@ -59,9 +59,9 @@ export class FileSystemStorageClient implements storage.StorageClient {
readonly logger?: CrawleeLogger;
readonly requestQueueAccess: 'single' | 'shared';
- readonly keyValueStoreCache: KeyValueStoreClient[] = [];
- readonly datasetClientCache: DatasetClient[] = [];
- readonly requestQueueCache: RequestQueueClient[] = [];
+ readonly keyValueStoreBackendCache: KeyValueStoreBackend[] = [];
+ readonly datasetBackendCache: DatasetBackend[] = [];
+ readonly requestQueueBackendCache: RequestQueueBackend[] = [];
constructor(options: FileSystemStorageOptions) {
s.object({
@@ -80,11 +80,11 @@ export class FileSystemStorageClient implements storage.StorageClient {
/**
* Return a cache key that includes the resolved storage directory, so that two
- * `FileSystemStorageClient` instances pointing at different directories get separate cache
+ * `FileSystemStorageBackend` instances pointing at different directories get separate cache
* partitions, by including the storage directory in the cache key.
*/
- getStorageClientCacheKey(): string {
- return `FileSystemStorageClient:${resolve(this.localDataDirectory)}`;
+ getStorageBackendCacheKey(): string {
+ return `FileSystemStorageBackend:${resolve(this.localDataDirectory)}`;
}
private static resolveStorageKey(options: { id?: string; name?: string; alias?: string }): {
@@ -100,11 +100,11 @@ export class FileSystemStorageClient implements storage.StorageClient {
return { id: options.id, name: options.name, alias: options.alias, cacheKey };
}
- async createDatasetClient(options: storage.CreateDatasetClientOptions = {}): Promise<storage.DatasetClient> {
- const { id, name, alias, cacheKey } = FileSystemStorageClient.resolveStorageKey(options);
+ async createDatasetBackend(options: storage.CreateDatasetBackendOptions = {}): Promise<storage.DatasetBackend> {
+ const { id, name, alias, cacheKey } = FileSystemStorageBackend.resolveStorageKey(options);
if (cacheKey) {
- const found = this.datasetClientCache.find(
+ const found = this.datasetBackendCache.find(
(store) =>
store.id === cacheKey ||
store.name?.toLowerCase() === cacheKey.toLowerCase() ||
@@ -115,25 +115,25 @@ export class FileSystemStorageClient implements storage.StorageClient {
}
}
- const nativeClient = await NativeDatasetClient.open(id, name, alias, this.localDataDirectory);
- const newStore = await DatasetClient.create({
+ const nativeBackend = await NativeDatasetBackend.open(id, name, alias, this.localDataDirectory);
+ const newStore = await DatasetBackend.create({
name: alias ? undefined : (name ?? cacheKey),
cacheKey: cacheKey ?? '',
- nativeClient,
+ nativeBackend,
logger: this.logger,
});
- this.datasetClientCache.push(newStore);
+ this.datasetBackendCache.push(newStore);
return newStore;
}
- async createKeyValueStoreClient(
- options: storage.CreateKeyValueStoreClientOptions = {},
- ): Promise<storage.KeyValueStoreClient> {
- const { id, name, alias, cacheKey } = FileSystemStorageClient.resolveStorageKey(options);
+ async createKeyValueStoreBackend(
+ options: storage.CreateKeyValueStoreBackendOptions = {},
+ ): Promise<storage.KeyValueStoreBackend> {
+ const { id, name, alias, cacheKey } = FileSystemStorageBackend.resolveStorageKey(options);
if (cacheKey) {
- const found = this.keyValueStoreCache.find(
+ const found = this.keyValueStoreBackendCache.find(
(store) =>
store.id === cacheKey ||
store.name?.toLowerCase() === cacheKey.toLowerCase() ||
@@ -144,25 +144,25 @@ export class FileSystemStorageClient implements storage.StorageClient {
}
}
- const nativeClient = await NativeKeyValueStoreClient.open(id, name, alias, this.localDataDirectory);
- const newStore = await KeyValueStoreClient.create({
+ const nativeBackend = await NativeKeyValueStoreBackend.open(id, name, alias, this.localDataDirectory);
+ const newStore = await KeyValueStoreBackend.create({
name: alias ? undefined : (name ?? cacheKey),
cacheKey: cacheKey ?? '',
- nativeClient,
+ nativeBackend,
logger: this.logger,
});
- this.keyValueStoreCache.push(newStore);
+ this.keyValueStoreBackendCache.push(newStore);
return newStore;
}
- async createRequestQueueClient(
- options: storage.CreateRequestQueueClientOptions = {},
- ): Promise<storage.RequestQueueClient> {
- const { id, name, alias, cacheKey } = FileSystemStorageClient.resolveStorageKey(options);
+ async createRequestQueueBackend(
+ options: storage.CreateRequestQueueBackendOptions = {},
+ ): Promise<storage.RequestQueueBackend> {
+ const { id, name, alias, cacheKey } = FileSystemStorageBackend.resolveStorageKey(options);
if (cacheKey) {
- const found = this.requestQueueCache.find(
+ const found = this.requestQueueBackendCache.find(
(queue) =>
queue.id === cacheKey ||
queue.name?.toLowerCase() === cacheKey.toLowerCase() ||
@@ -173,7 +173,7 @@ export class FileSystemStorageClient implements storage.StorageClient {
}
}
- const nativeClient = await NativeRequestQueueClient.open(
+ const nativeBackend = await NativeRequestQueueBackend.open(
id,
name,
alias,
@@ -182,32 +182,32 @@ export class FileSystemStorageClient implements storage.StorageClient {
undefined,
this.requestQueueAccess,
);
- const newStore = await RequestQueueClient.create({
+ const newStore = await RequestQueueBackend.create({
name: alias ? undefined : (name ?? cacheKey),
cacheKey: cacheKey ?? '',
- nativeClient,
+ nativeBackend,
logger: this.logger,
});
- this.requestQueueCache.push(newStore);
+ this.requestQueueBackendCache.push(newStore);
return newStore;
}
async storageExists(id: string, type: 'Dataset' | 'KeyValueStore' | 'RequestQueue'): Promise<boolean> {
- let clients: (KeyValueStoreClient | DatasetClient | RequestQueueClient)[];
+ let backends: (KeyValueStoreBackend | DatasetBackend | RequestQueueBackend)[];
let baseDir: string;
switch (type) {
case 'Dataset':
- clients = this.datasetClientCache;
+ backends = this.datasetBackendCache;
baseDir = this.datasetsDirectory;
break;
case 'KeyValueStore':
- clients = this.keyValueStoreCache;
+ backends = this.keyValueStoreBackendCache;
baseDir = this.keyValueStoresDirectory;
break;
case 'RequestQueue':
- clients = this.requestQueueCache;
+ backends = this.requestQueueBackendCache;
baseDir = this.requestQueuesDirectory;
break;
default:
@@ -215,7 +215,7 @@ export class FileSystemStorageClient implements storage.StorageClient {
}
// Check the in-memory cache by actual storage ID first.
- if (clients.some((store) => store.id === id)) {
+ if (backends.some((store) => store.id === id)) {
return true;
}
@@ -228,7 +228,7 @@ export class FileSystemStorageClient implements storage.StorageClient {
// has a matching directory. We therefore read the real id from the metadata and only report
// existence when it equals the queried string. This matches upstream PR #3800/#3808 and
// prevents a named storage from being re-resolved as `{ id: name }` on a subsequent run.
- const resolvedId = await FileSystemStorageClient.resolveStorageIdOnDisk(baseDir, id);
+ const resolvedId = await FileSystemStorageBackend.resolveStorageIdOnDisk(baseDir, id);
return resolvedId === id;
}
@@ -246,7 +246,7 @@ export class FileSystemStorageClient implements storage.StorageClient {
): Promise<string | undefined> {
// Directory named exactly after the string: return its real (metadata) id, which may differ
// from the string when the string is a name rather than an id.
- const directId = await FileSystemStorageClient.readMetadataId(resolve(baseDirectory, entryNameOrId));
+ const directId = await FileSystemStorageBackend.readMetadataId(resolve(baseDirectory, entryNameOrId));
if (directId !== undefined) {
return directId;
}
@@ -264,7 +264,7 @@ export class FileSystemStorageClient implements storage.StorageClient {
continue;
}
- const metadataId = await FileSystemStorageClient.readMetadataId(resolve(baseDirectory, directory.name));
+ const metadataId = await FileSystemStorageBackend.readMetadataId(resolve(baseDirectory, directory.name));
if (metadataId === entryNameOrId) {
return metadataId;
}
@@ -293,14 +293,14 @@ export class FileSystemStorageClient implements storage.StorageClient {
async purge(): Promise<void> {
// Resolve the default stores up front so leftover on-disk records are purged even when the
// store has not been opened in this process yet (e.g. a fresh run over a pre-existing
- // directory). Opening caches the client, so the subsequent purge operates on a real client.
+ // directory). Opening caches the backend, so the subsequent purge operates on a real backend.
// The default store is opened via the internal `__default__` alias (see resolveStorageIdentifier
// in @crawlee/core), which resolves to the `default` cache key — match that here so we purge the
- // very client the default open would return rather than creating a divergent one.
+ // very backend the default open would return rather than creating a divergent one.
const [defaultKeyValueStore, defaultDataset, defaultRequestQueue] = await Promise.all([
- this.createKeyValueStoreClient({ alias: '__default__' }) as Promise<KeyValueStoreClient>,
- this.createDatasetClient({ alias: '__default__' }) as Promise<DatasetClient>,
- this.createRequestQueueClient({ alias: '__default__' }) as Promise<RequestQueueClient>,
+ this.createKeyValueStoreBackend({ alias: '__default__' }) as Promise<KeyValueStoreBackend>,
+ this.createDatasetBackend({ alias: '__default__' }) as Promise<DatasetBackend>,
+ this.createRequestQueueBackend({ alias: '__default__' }) as Promise<RequestQueueBackend>,
]);
await Promise.all([
@@ -318,6 +318,6 @@ export class FileSystemStorageClient implements storage.StorageClient {
* are not stuck (until their lock expires) for the next consumer of the same on-disk queue.
*/
async teardown(): Promise<void> {
- await Promise.all(this.requestQueueCache.map(async (queue) => queue.persistState()));
+ await Promise.all(this.requestQueueBackendCache.map(async (queue) => queue.persistState()));
}
}
diff --git a/packages/fs-storage/src/resource-clients/cached-id-client.ts b/packages/fs-storage/src/resource-clients/cached-id-client.ts
index fb8dd5e5a..ac0a2e200 100644
--- a/packages/fs-storage/src/resource-clients/cached-id-client.ts
+++ b/packages/fs-storage/src/resource-clients/cached-id-client.ts
@@ -1,7 +1,7 @@
/**
- * Shared base for the file-system resource clients. The native id is read once from the native
- * metadata at construction time (in each client's `create()`), so the synchronous `id` getter —
- * required by `FileSystemStorageClient.storageExists` and the cache lookups — does not have to await.
+ * Shared base for the file-system resource backends. The native id is read once from the native
+ * metadata at construction time (in each backend's `create()`), so the synchronous `id` getter —
+ * required by `FileSystemStorageBackend.storageExists` and the cache lookups — does not have to await.
*/
export abstract class CachedIdClient {
/** The storage id assigned by the native client. Set once by the subclass `create()`. */
diff --git a/packages/fs-storage/src/resource-clients/dataset.ts b/packages/fs-storage/src/resource-clients/dataset.ts
index c999b4100..88dd62781 100644
--- a/packages/fs-storage/src/resource-clients/dataset.ts
+++ b/packages/fs-storage/src/resource-clients/dataset.ts
@@ -2,7 +2,7 @@ import type * as storage from '@crawlee/types';
import type { CrawleeLogger, Dictionary } from '@crawlee/types';
import { s } from '@sapphire/shapeshift';
-import type { FileSystemDatasetClient as NativeFileSystemDatasetClient } from '@crawlee/fs-storage-native';
+import type { FileSystemDatasetClient as NativeFileSystemDatasetBackend } from '@crawlee/fs-storage-native';
import { CachedIdClient } from './cached-id-client.js';
@@ -16,72 +16,72 @@ import { CachedIdClient } from './cached-id-client.js';
*/
const UNSUPPORTED_GET_DATA_OPTIONS = ['clean', 'fields', 'omit', 'skipHidden', 'skipEmpty'] as const;
-export interface DatasetClientOptions {
+export interface DatasetBackendOptions {
/** The user-facing storage name, or `undefined` for unnamed (alias / default) storages. */
name?: string;
/**
- * The key used for cache lookup in {@link FileSystemStorageClient}. For named storages this equals
+ * The key used for cache lookup in {@link FileSystemStorageBackend}. For named storages this equals
* the name; for alias (unnamed) storages it is the alias string. Falls back to the storage id.
*/
cacheKey: string;
- nativeClient: NativeFileSystemDatasetClient;
+ nativeBackend: NativeFileSystemDatasetBackend;
logger?: CrawleeLogger;
}
/**
- * A file-system dataset client backed by the native `@crawlee/fs-storage-native` Rust extension.
+ * A file-system dataset backend backed by the native `@crawlee/fs-storage-native` Rust extension.
*
* This class is a thin adapter: it forwards each operation to the native client (which owns the
* on-disk format, timestamps and item counting) and converts results into the shapes expected by
* the `@crawlee/types` interfaces.
*/
-export class DatasetClient<Data extends Dictionary = Dictionary>
+export class DatasetBackend<Data extends Dictionary = Dictionary>
extends CachedIdClient
- implements storage.DatasetClient<Data>
+ implements storage.DatasetBackend<Data>
{
readonly name?: string;
readonly cacheKey: string;
- private readonly nativeClient: NativeFileSystemDatasetClient;
+ private readonly nativeBackend: NativeFileSystemDatasetBackend;
private readonly logger?: CrawleeLogger;
- constructor(options: DatasetClientOptions) {
+ constructor(options: DatasetBackendOptions) {
super();
this.name = options.name;
this.cacheKey = options.cacheKey;
- this.nativeClient = options.nativeClient;
+ this.nativeBackend = options.nativeBackend;
this.logger = options.logger;
}
get datasetDirectory(): string {
- return this.nativeClient.pathToDataset;
+ return this.nativeBackend.pathToDataset;
}
static async create<Data extends Dictionary = Dictionary>(
- options: DatasetClientOptions,
- ): Promise<DatasetClient<Data>> {
- const client = new DatasetClient<Data>(options);
- client._cachedId = (await options.nativeClient.getMetadata()).id;
- return client;
+ options: DatasetBackendOptions,
+ ): Promise<DatasetBackend<Data>> {
+ const backend = new DatasetBackend<Data>(options);
+ backend._cachedId = (await options.nativeBackend.getMetadata()).id;
+ return backend;
}
async getMetadata(): Promise<storage.DatasetInfo> {
- return this.nativeClient.getMetadata();
+ return this.nativeBackend.getMetadata();
}
async drop(): Promise<void> {
- await this.nativeClient.dropStorage();
+ await this.nativeBackend.dropStorage();
}
async purge(): Promise<void> {
- await this.nativeClient.purge();
+ await this.nativeBackend.purge();
}
async pushData(items: Data[]): Promise<void> {
- await this.nativeClient.pushData(items);
+ await this.nativeBackend.pushData(items);
}
- async getData(options: storage.DatasetClientListOptions = {}): Promise<storage.PaginatedList<Data>> {
+ async getData(options: storage.DatasetBackendListOptions = {}): Promise<storage.PaginatedList<Data>> {
const passedOptions = options as Record<string, unknown>;
const ignored = UNSUPPORTED_GET_DATA_OPTIONS.filter((key) => passedOptions[key] !== undefined);
if (ignored.length > 0) {
@@ -99,7 +99,7 @@ export class DatasetClient<Data extends Dictionary = Dictionary>
})
.parse(options);
- const page = await this.nativeClient.getData(offset ?? 0, limit, desc ?? false, false);
+ const page = await this.nativeBackend.getData(offset ?? 0, limit, desc ?? false, false);
return {
count: page.count,
diff --git a/packages/fs-storage/src/resource-clients/key-value-store.ts b/packages/fs-storage/src/resource-clients/key-value-store.ts
index fd7af062e..6b38e985a 100644
--- a/packages/fs-storage/src/resource-clients/key-value-store.ts
+++ b/packages/fs-storage/src/resource-clients/key-value-store.ts
@@ -5,7 +5,7 @@ import type { CrawleeLogger } from '@crawlee/types';
import { s } from '@sapphire/shapeshift';
import type {
- FileSystemKeyValueStoreClient as NativeFileSystemKeyValueStoreClient,
+ FileSystemKeyValueStoreClient as NativeFileSystemKeyValueStoreBackend,
ListBareFallback,
} from '@crawlee/fs-storage-native';
import { isStream } from '../utils.js';
@@ -61,71 +61,71 @@ const BARE_FILE_LOGICAL_KEYS = new Map(
),
);
-export interface KeyValueStoreClientOptions {
+export interface KeyValueStoreBackendOptions {
/** The user-facing storage name, or `undefined` for unnamed (alias / default) storages. */
name?: string;
/**
- * The key used for cache lookup in {@link FileSystemStorageClient}. For named storages this equals
+ * The key used for cache lookup in {@link FileSystemStorageBackend}. For named storages this equals
* the name; for alias (unnamed) storages it is the alias string. Falls back to the storage id.
*/
cacheKey: string;
- nativeClient: NativeFileSystemKeyValueStoreClient;
+ nativeBackend: NativeFileSystemKeyValueStoreBackend;
logger?: CrawleeLogger;
}
/**
- * A file-system key-value store client backed by the native `@crawlee/fs-storage-native` Rust
+ * A file-system key-value store backend backed by the native `@crawlee/fs-storage-native` Rust
* extension.
*
* This adapter is a plain byte transport: values are written and read verbatim as `Buffer`s with a
* content type carried alongside them. Serializing arbitrary values into bytes and parsing them back
- * is the {@apilink KeyValueStore} frontend codec's job, not this client's.
+ * is the {@apilink KeyValueStore} frontend codec's job, not this backend's.
*/
-export class KeyValueStoreClient extends CachedIdClient implements storage.KeyValueStoreClient {
+export class KeyValueStoreBackend extends CachedIdClient implements storage.KeyValueStoreBackend {
readonly name?: string;
readonly cacheKey: string;
- private readonly nativeClient: NativeFileSystemKeyValueStoreClient;
+ private readonly nativeBackend: NativeFileSystemKeyValueStoreBackend;
- constructor(options: KeyValueStoreClientOptions) {
+ constructor(options: KeyValueStoreBackendOptions) {
super();
this.name = options.name;
this.cacheKey = options.cacheKey;
- this.nativeClient = options.nativeClient;
+ this.nativeBackend = options.nativeBackend;
}
get keyValueStoreDirectory(): string {
- return this.nativeClient.pathToKvs;
+ return this.nativeBackend.pathToKvs;
}
- static async create(options: KeyValueStoreClientOptions): Promise<KeyValueStoreClient> {
- const client = new KeyValueStoreClient(options);
- client._cachedId = (await options.nativeClient.getMetadata()).id;
- return client;
+ static async create(options: KeyValueStoreBackendOptions): Promise<KeyValueStoreBackend> {
+ const backend = new KeyValueStoreBackend(options);
+ backend._cachedId = (await options.nativeBackend.getMetadata()).id;
+ return backend;
}
async getMetadata(): Promise<storage.KeyValueStoreInfo> {
- return this.nativeClient.getMetadata();
+ return this.nativeBackend.getMetadata();
}
async drop(): Promise<void> {
- await this.nativeClient.dropStorage();
+ await this.nativeBackend.dropStorage();
}
async purge(): Promise<void> {
- await this.nativeClient.purge();
+ await this.nativeBackend.purge();
}
/**
* Remove every record from the store except the run input. Used by
- * {@link FileSystemStorageClient.purge} to clean the default key-value store at the start of a run
+ * {@link FileSystemStorageBackend.purge} to clean the default key-value store at the start of a run
* while preserving the run's input, matching the historical file-system storage behavior.
*
* The native `purge` keep-list matches by exact key with no extension globbing, so we pass every
* filename the input might live under (`INPUT`, `INPUT.json`, `INPUT.txt`, `INPUT.bin`).
*/
async purgeExceptInput(): Promise<void> {
- await this.nativeClient.purge(BARE_FILE_FALLBACKS.flatMap(({ extension }) => `INPUT${extension}`));
+ await this.nativeBackend.purge(BARE_FILE_FALLBACKS.flatMap(({ extension }) => `INPUT${extension}`));
}
async listKeys(options: storage.KeyValueStoreListKeysOptions = {}): Promise<storage.KeyValueStoreListKeysResult> {
@@ -142,7 +142,7 @@ export class KeyValueStoreClient extends CachedIdClient implements storage.KeyVa
// everything it needs off the filesystem index — no per-file reads — so this stays cheap.
// The native `listKeys` already returns a self-describing page (items + pagination cursors)
// matching the `KeyValueStoreListKeysResult` contract, so we only post-process the items.
- const page = await this.nativeClient.listKeys(exclusiveStartKey, limit, prefix, LIST_BARE_FALLBACKS);
+ const page = await this.nativeBackend.listKeys(exclusiveStartKey, limit, prefix, LIST_BARE_FALLBACKS);
const presentKeys = new Set(page.items.map((record) => record.key));
@@ -183,7 +183,7 @@ export class KeyValueStoreClient extends CachedIdClient implements storage.KeyVa
if (resolvedKey === undefined) {
return undefined;
}
- return (await this.nativeClient.getPublicUrl(resolvedKey)) ?? undefined;
+ return (await this.nativeBackend.getPublicUrl(resolvedKey)) ?? undefined;
}
/**
@@ -202,8 +202,8 @@ export class KeyValueStoreClient extends CachedIdClient implements storage.KeyVa
const fallbacks = this.bareFallbacksFor(key);
const record = fallbacks
- ? await this.nativeClient.resolveValue(key, fallbacks)
- : await this.nativeClient.getValue(key);
+ ? await this.nativeBackend.resolveValue(key, fallbacks)
+ : await this.nativeBackend.getValue(key);
if (record) {
return {
@@ -217,7 +217,7 @@ export class KeyValueStoreClient extends CachedIdClient implements storage.KeyVa
}
async setValue(record: storage.KeyValueStoreInputRecord): Promise<void> {
- // By the time a value reaches the client the frontend (KeyValueStore codec) has already
+ // By the time a value reaches the backend the frontend (KeyValueStore codec) has already
// serialized it: non-bytes become a `string`, everything else is a `Buffer`/typed array or a
// stream. So we only accept those shapes here — there is no JSON inference or `String(value)`
// coercion left to do.
@@ -236,7 +236,7 @@ export class KeyValueStoreClient extends CachedIdClient implements storage.KeyVa
}).parse(record);
const { key, value } = record;
- // The frontend resolves the content type before it reaches the client; this client is a plain
+ // The frontend resolves the content type before it reaches the backend; this backend is a plain
// byte transport and does not infer content types.
const contentType = record.contentType ?? 'application/octet-stream';
@@ -244,7 +244,7 @@ export class KeyValueStoreClient extends CachedIdClient implements storage.KeyVa
// consumes a Web `ReadableStream`, so convert the Node `Readable` we get from the frontend.
if (isStream(value)) {
const webStream = Readable.toWeb(value as Readable) as ReadableStream<Uint8Array>;
- await this.nativeClient.setValueStream(key, webStream, contentType);
+ await this.nativeBackend.setValueStream(key, webStream, contentType);
return;
}
@@ -257,12 +257,12 @@ export class KeyValueStoreClient extends CachedIdClient implements storage.KeyVa
? Buffer.from(value.buffer, value.byteOffset, value.byteLength)
: Buffer.from(value as string);
- await this.nativeClient.setValue(key, buffer, contentType);
+ await this.nativeBackend.setValue(key, buffer, contentType);
}
async deleteValue(key: string): Promise<void> {
s.string().parse(key);
- await this.nativeClient.deleteValue(key);
+ await this.nativeBackend.deleteValue(key);
}
/**
@@ -277,13 +277,13 @@ export class KeyValueStoreClient extends CachedIdClient implements storage.KeyVa
const fallbacks = this.bareFallbacksFor(key);
if (fallbacks) {
return (
- (await this.nativeClient.resolveExistingKey(
+ (await this.nativeBackend.resolveExistingKey(
key,
fallbacks.map(({ extension }) => extension),
)) ?? undefined
);
}
- return (await this.nativeClient.recordExists(key)) ? key : undefined;
+ return (await this.nativeBackend.recordExists(key)) ? key : undefined;
}
/**
diff --git a/packages/fs-storage/src/resource-clients/request-queue.ts b/packages/fs-storage/src/resource-clients/request-queue.ts
index fb8a86fe5..ac26cde11 100644
--- a/packages/fs-storage/src/resource-clients/request-queue.ts
+++ b/packages/fs-storage/src/resource-clients/request-queue.ts
@@ -2,7 +2,7 @@ import type * as storage from '@crawlee/types';
import type { CrawleeLogger } from '@crawlee/types';
import { s } from '@sapphire/shapeshift';
-import type { FileSystemRequestQueueClient as NativeFileSystemRequestQueueClient } from '@crawlee/fs-storage-native';
+import type { FileSystemRequestQueueClient as NativeFileSystemRequestQueueBackend } from '@crawlee/fs-storage-native';
import { CachedIdClient } from './cached-id-client.js';
@@ -40,46 +40,46 @@ const requestOptionsShape = s.object({
forefront: s.boolean().optional(),
});
-export interface RequestQueueClientOptions {
+export interface RequestQueueBackendOptions {
/** The user-facing storage name, or `undefined` for unnamed (alias / default) storages. */
name?: string;
/**
- * The key used for cache lookup in {@link FileSystemStorageClient}. For named storages this equals
+ * The key used for cache lookup in {@link FileSystemStorageBackend}. For named storages this equals
* the name; for alias (unnamed) storages it is the alias string. Falls back to the storage id.
*/
cacheKey: string;
- nativeClient: NativeFileSystemRequestQueueClient;
+ nativeBackend: NativeFileSystemRequestQueueBackend;
logger?: CrawleeLogger;
}
/**
- * A file-system request queue client backed by the native `@crawlee/fs-storage-native` Rust
+ * A file-system request queue backend backed by the native `@crawlee/fs-storage-native` Rust
* extension.
*
* Request ordering, in-progress locking and state persistence are all owned by the native client.
* This adapter forwards each operation and converts result shapes to the `@crawlee/types` interfaces.
*/
-export class RequestQueueClient extends CachedIdClient implements storage.RequestQueueClient {
+export class RequestQueueBackend extends CachedIdClient implements storage.RequestQueueBackend {
readonly name?: string;
readonly cacheKey: string;
- private readonly nativeClient: NativeFileSystemRequestQueueClient;
+ private readonly nativeBackend: NativeFileSystemRequestQueueBackend;
- constructor(options: RequestQueueClientOptions) {
+ constructor(options: RequestQueueBackendOptions) {
super();
this.name = options.name;
this.cacheKey = options.cacheKey;
- this.nativeClient = options.nativeClient;
+ this.nativeBackend = options.nativeBackend;
}
get requestQueueDirectory(): string {
- return this.nativeClient.pathToRq;
+ return this.nativeBackend.pathToRq;
}
- static async create(options: RequestQueueClientOptions): Promise<RequestQueueClient> {
- const client = new RequestQueueClient(options);
- client._cachedId = (await options.nativeClient.getMetadata()).id;
- return client;
+ static async create(options: RequestQueueBackendOptions): Promise<RequestQueueBackend> {
+ const backend = new RequestQueueBackend(options);
+ backend._cachedId = (await options.nativeBackend.getMetadata()).id;
+ return backend;
}
/**
@@ -87,19 +87,19 @@ export class RequestQueueClient extends CachedIdClient implements storage.Reques
* available again.
*/
async setExpectedRequestProcessingTimeSecs(secs: number): Promise<void> {
- await this.nativeClient.setExpectedRequestProcessingTime(secs);
+ await this.nativeBackend.setExpectedRequestProcessingTime(secs);
}
async getMetadata(): Promise<storage.RequestQueueInfo> {
- return this.nativeClient.getMetadata();
+ return this.nativeBackend.getMetadata();
}
async drop(): Promise<void> {
- await this.nativeClient.dropStorage();
+ await this.nativeBackend.dropStorage();
}
async purge(): Promise<void> {
- await this.nativeClient.purge();
+ await this.nativeBackend.purge();
}
async addBatchOfRequests(
@@ -109,7 +109,7 @@ export class RequestQueueClient extends CachedIdClient implements storage.Reques
batchRequestShapeWithoutId.parse(requests);
requestOptionsShape.parse(options);
- const response = await this.nativeClient.addBatchOfRequests(
+ const response = await this.nativeBackend.addBatchOfRequests(
requests.map((request) => plainifyRequest(request)),
options.forefront ?? false,
);
@@ -128,16 +128,16 @@ export class RequestQueueClient extends CachedIdClient implements storage.Reques
// The native client tags requests with an internal `orderNo`; it's harmless to leak, so we
// hand the request back as-is rather than copying it just to drop one undeclared property.
// The native client already returns `undefined` for a missing request, matching this contract.
- return (await this.nativeClient.getRequest(uniqueKey)) as storage.UpdateRequestSchema | undefined;
+ return (await this.nativeBackend.getRequest(uniqueKey)) as storage.UpdateRequestSchema | undefined;
}
async fetchNextRequest(): Promise<storage.UpdateRequestSchema | undefined> {
- return (await this.nativeClient.fetchNextRequest()) as storage.UpdateRequestSchema | undefined;
+ return (await this.nativeBackend.fetchNextRequest()) as storage.UpdateRequestSchema | undefined;
}
async markRequestAsHandled(request: storage.UpdateRequestSchema): Promise<storage.QueueOperationInfo | undefined> {
requestShape.parse(request);
- return (await this.nativeClient.markRequestAsHandled(plainifyRequest(request))) ?? undefined;
+ return (await this.nativeBackend.markRequestAsHandled(plainifyRequest(request))) ?? undefined;
}
async reclaimRequest(
@@ -147,24 +147,24 @@ export class RequestQueueClient extends CachedIdClient implements storage.Reques
requestShape.parse(request);
requestOptionsShape.parse(options);
return (
- (await this.nativeClient.reclaimRequest(plainifyRequest(request), options.forefront ?? false)) ?? undefined
+ (await this.nativeBackend.reclaimRequest(plainifyRequest(request), options.forefront ?? false)) ?? undefined
);
}
async isEmpty(): Promise<boolean> {
- return this.nativeClient.isEmpty();
+ return this.nativeBackend.isEmpty();
}
async isFinished(): Promise<boolean> {
- return this.nativeClient.isFinished();
+ return this.nativeBackend.isFinished();
}
/**
* Persist the native client's in-memory state to disk. Called by
- * {@link FileSystemStorageClient.teardown} so that fetched-but-unhandled requests are not stuck
+ * {@link FileSystemStorageBackend.teardown} so that fetched-but-unhandled requests are not stuck
* for the next consumer of the same on-disk queue.
*/
async persistState(): Promise<void> {
- await this.nativeClient.persistState();
+ await this.nativeBackend.persistState();
}
}
diff --git a/packages/fs-storage/test/async-iteration.test.ts b/packages/fs-storage/test/async-iteration.test.ts
index 5cbf0cc69..3374e5616 100644
--- a/packages/fs-storage/test/async-iteration.test.ts
+++ b/packages/fs-storage/test/async-iteration.test.ts
@@ -1,12 +1,12 @@
import { rm } from 'node:fs/promises';
import path from 'node:path';
-import { FileSystemStorageClient } from '@crawlee/fs-storage';
-import type { DatasetClient, KeyValueStoreClient } from '@crawlee/types';
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
+import type { DatasetBackend, KeyValueStoreBackend } from '@crawlee/types';
describe('Async iteration support', () => {
const localDataDirectory = path.resolve(__dirname, './tmp/async-iteration');
- const storage = new FileSystemStorageClient({
+ const storage = new FileSystemStorageBackend({
localDataDirectory,
});
@@ -16,10 +16,10 @@ describe('Async iteration support', () => {
describe('Dataset.getData', () => {
const elements = Array.from({ length: 25 }, (_, i) => ({ index: i }));
- let dataset: DatasetClient<{ index: number }>;
+ let dataset: DatasetBackend<{ index: number }>;
beforeAll(async () => {
- dataset = (await storage.createDatasetClient({ name: 'async-iteration-dataset' })) as DatasetClient<{
+ dataset = (await storage.createDatasetBackend({ name: 'async-iteration-dataset' })) as DatasetBackend<{
index: number;
}>;
await dataset.pushData(elements);
@@ -65,10 +65,10 @@ describe('Async iteration support', () => {
describe('KeyValueStore.listKeys', () => {
const keys = Array.from({ length: 25 }, (_, i) => `key-${String(i).padStart(2, '0')}`);
- let kvStore: KeyValueStoreClient;
+ let kvStore: KeyValueStoreBackend;
beforeAll(async () => {
- kvStore = await storage.createKeyValueStoreClient({ name: 'async-iteration-kvs' });
+ kvStore = await storage.createKeyValueStoreBackend({ name: 'async-iteration-kvs' });
for (const key of keys) {
// The client is a byte transport: values arrive already serialized from the frontend
diff --git a/packages/fs-storage/test/fs-fallback.test.ts b/packages/fs-storage/test/fs-fallback.test.ts
index e8ecb2177..a3c2f7436 100644
--- a/packages/fs-storage/test/fs-fallback.test.ts
+++ b/packages/fs-storage/test/fs-fallback.test.ts
@@ -2,12 +2,12 @@ import { randomUUID } from 'node:crypto';
import { mkdir, rm, writeFile } from 'node:fs/promises';
import { resolve } from 'node:path';
-import { FileSystemStorageClient } from '@crawlee/fs-storage';
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
import type { KeyValueStoreRecord } from '@crawlee/types';
// The storage is backed by the native `@crawlee/fs-storage-native` extension, which only serves
// key-value records it has written itself (tracked via per-record metadata sidecars). The
-// `KeyValueStoreClient` adapter layers a fallback on top so that value files placed into the store
+// `KeyValueStoreBackend` adapter layers a fallback on top so that value files placed into the store
// directory out-of-band — e.g. a hand-written or platform-provided `INPUT.json` — are still readable.
// These tests pin both the store-identity metadata fallback and that bare-file fallback.
//
@@ -20,7 +20,7 @@ import type { KeyValueStoreRecord } from '@crawlee/types';
// lookup keeps resolving the same bare file.
describe('fallback to fs for reading', () => {
const tmpLocation = resolve(import.meta.dirname, './tmp/fs-fallback');
- const storage = new FileSystemStorageClient({
+ const storage = new FileSystemStorageBackend({
localDataDirectory: tmpLocation,
});
@@ -76,7 +76,7 @@ describe('fallback to fs for reading', () => {
});
test('reads store identity from the on-disk metadata, and a bare INPUT.json value', async () => {
- const defaultStore = await storage.createKeyValueStoreClient({ name: 'default' });
+ const defaultStore = await storage.createKeyValueStoreBackend({ name: 'default' });
const defaultStoreInfo = await defaultStore.getMetadata();
expect(defaultStoreInfo.name).toEqual('default');
@@ -93,7 +93,7 @@ describe('fallback to fs for reading', () => {
});
test('reads a bare INPUT.json even with no store metadata present', async () => {
- const otherStore = await storage.createKeyValueStoreClient({ name: 'other' });
+ const otherStore = await storage.createKeyValueStoreBackend({ name: 'other' });
// Byte transport: raw bytes out, parsing is the frontend's job.
const input = await otherStore.getValue('INPUT');
@@ -105,13 +105,13 @@ describe('fallback to fs for reading', () => {
});
test('a store with no data on disk is still accessible after creation', async () => {
- const default2Store = await storage.createKeyValueStoreClient({ name: 'default_2' });
+ const default2Store = await storage.createKeyValueStoreBackend({ name: 'default_2' });
const info = await default2Store.getMetadata();
expect(info.name).toEqual('default_2');
});
test('loads a value file with no extension as raw bytes with a generic content type', async () => {
- const noExtStore = await storage.createKeyValueStoreClient({ name: 'no-ext' });
+ const noExtStore = await storage.createKeyValueStoreBackend({ name: 'no-ext' });
// Byte transport: the no-extension fallback returns raw bytes. With no extension to infer a
// content type from, the native client reports the generic `application/octet-stream`.
@@ -124,7 +124,7 @@ describe('fallback to fs for reading', () => {
});
test('returns an invalid-JSON bare value file verbatim', async () => {
- const invalidJsonStore = await storage.createKeyValueStoreClient({ name: 'invalid-json' });
+ const invalidJsonStore = await storage.createKeyValueStoreBackend({ name: 'invalid-json' });
// Byte transport: the client no longer validates parseability. Malformed JSON is returned
// verbatim as raw bytes; parsing (and any resulting error) is the KeyValueStore frontend's job.
@@ -137,7 +137,7 @@ describe('fallback to fs for reading', () => {
});
test('bare files are visible to recordExists, getPublicUrl, and listKeys', async () => {
- const otherStore = await storage.createKeyValueStoreClient({ name: 'other' });
+ const otherStore = await storage.createKeyValueStoreBackend({ name: 'other' });
expect(await otherStore.recordExists('INPUT')).toBe(true);
expect(await otherStore.recordExists('does-not-exist')).toBe(false);
@@ -151,7 +151,7 @@ describe('fallback to fs for reading', () => {
});
test('a listed bare key round-trips through getValue / recordExists / getPublicUrl', async () => {
- const otherStore = await storage.createKeyValueStoreClient({ name: 'other' });
+ const otherStore = await storage.createKeyValueStoreBackend({ name: 'other' });
// The key `listKeys` reports for a bare file must be readable back verbatim, while the logical
// `INPUT` lookup keeps resolving the same bare file (matching how Crawlee reads run input).
@@ -172,7 +172,7 @@ describe('fallback to fs for reading', () => {
});
test('the bare-file fallback is scoped to INPUT: a non-INPUT bare file is ignored', async () => {
- const nonInputStore = await storage.createKeyValueStoreClient({ name: 'non-input' });
+ const nonInputStore = await storage.createKeyValueStoreBackend({ name: 'non-input' });
// `some-key.json` sits on disk with no metadata sidecar. Only `INPUT` keys probe bare files,
// so this is invisible to every read path: it has no tracked record, and the `.json` extension
@@ -187,7 +187,7 @@ describe('fallback to fs for reading', () => {
});
test('a tracked INPUT record shadows the bare INPUT.json variant in listKeys', async () => {
- const collisionStore = await storage.createKeyValueStoreClient({ name: 'input-collision' });
+ const collisionStore = await storage.createKeyValueStoreBackend({ name: 'input-collision' });
// Write a tracked `INPUT` record (value file + metadata sidecar), then drop a sidecar-less bare
// `INPUT.json` next to it. Both belong to the logical key `INPUT`; the tracked record wins, so
@@ -221,7 +221,7 @@ const BARE_VARIANTS = [
// variant lives in its own store so the logical-`INPUT` lookup resolves it unambiguously.
describe('run-input bare-file reachability (one variant per store)', () => {
const tmpLocation = resolve(import.meta.dirname, './tmp/fs-reachability-isolated');
- const storage = new FileSystemStorageClient({ localDataDirectory: tmpLocation });
+ const storage = new FileSystemStorageBackend({ localDataDirectory: tmpLocation });
const storeNameFor = (file: string) => `reach-${file.toLowerCase().replace('.', '-')}`;
@@ -247,7 +247,7 @@ describe('run-input bare-file reachability (one variant per store)', () => {
);
test.each(reachableKeys)('is reachable via %s', async (key) => {
- const store = await storage.createKeyValueStoreClient({ name: storeNameFor(file) });
+ const store = await storage.createKeyValueStoreBackend({ name: storeNameFor(file) });
expect(await store.getValue(key)).toStrictEqual<KeyValueStoreRecord>({
key,
@@ -259,7 +259,7 @@ describe('run-input bare-file reachability (one variant per store)', () => {
});
test.each(unreachableKeys)('is not reachable via %s', async (key) => {
- const store = await storage.createKeyValueStoreClient({ name: storeNameFor(file) });
+ const store = await storage.createKeyValueStoreBackend({ name: storeNameFor(file) });
expect(await store.getValue(key)).toBeUndefined();
expect(await store.recordExists(key)).toBe(false);
@@ -273,7 +273,7 @@ describe('run-input bare-file reachability (one variant per store)', () => {
// extensionless `INPUT`. This is what fails if literal-name probing ever widens to other extensions.
describe('run-input bare-file reachability (all variants in one store)', () => {
const tmpLocation = resolve(import.meta.dirname, './tmp/fs-reachability-shared');
- const storage = new FileSystemStorageClient({ localDataDirectory: tmpLocation });
+ const storage = new FileSystemStorageBackend({ localDataDirectory: tmpLocation });
beforeAll(async () => {
const dir = resolve(storage.keyValueStoresDirectory, 'all-variants');
@@ -290,7 +290,7 @@ describe('run-input bare-file reachability (all variants in one store)', () => {
test.each(BARE_VARIANTS)(
'the literal key $literalKey reads its own file, not a sibling',
async ({ literalKey, contentType, payload }) => {
- const store = await storage.createKeyValueStoreClient({ name: 'all-variants' });
+ const store = await storage.createKeyValueStoreBackend({ name: 'all-variants' });
expect(await store.getValue(literalKey)).toStrictEqual<KeyValueStoreRecord>({
key: literalKey,
@@ -301,7 +301,7 @@ describe('run-input bare-file reachability (all variants in one store)', () => {
);
test('the logical INPUT key resolves the extensionless file (first ladder match)', async () => {
- const store = await storage.createKeyValueStoreClient({ name: 'all-variants' });
+ const store = await storage.createKeyValueStoreBackend({ name: 'all-variants' });
const extensionless = BARE_VARIANTS.find((variant) => variant.file === 'INPUT')!;
expect(await store.getValue('INPUT')).toStrictEqual<KeyValueStoreRecord>({
diff --git a/packages/fs-storage/test/key-value-store/record-key-path-traversal.test.ts b/packages/fs-storage/test/key-value-store/record-key-path-traversal.test.ts
index 46385689e..d660b3cb2 100644
--- a/packages/fs-storage/test/key-value-store/record-key-path-traversal.test.ts
+++ b/packages/fs-storage/test/key-value-store/record-key-path-traversal.test.ts
@@ -1,11 +1,11 @@
import { rm } from 'node:fs/promises';
import { resolve } from 'node:path';
-import { FileSystemStorageClient } from '@crawlee/fs-storage';
-import { pathExists } from 'fs-extra/esm';
-
-import { waitTillWrittenToDisk } from '../__shared__.js';
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
+// On-disk file naming/encoding is the native `@crawlee/fs-storage-native` client's concern (see
+// `special-keys.test.ts`), so these tests only assert through the public API rather than reaching
+// for specific file paths on disk.
describe('record key path traversal', () => {
const tmpLocation = resolve(import.meta.dirname, './tmp/record-key-path-traversal');
@@ -13,34 +13,23 @@ describe('record key path traversal', () => {
await rm(tmpLocation, { force: true, recursive: true });
});
- const storage = new FileSystemStorageClient({
- localDataDirectory: tmpLocation,
- writeMetadata: true,
- });
+ const storage = new FileSystemStorageBackend({ localDataDirectory: tmpLocation });
test('setValue rejects a key that escapes the store directory', async () => {
- const client = await storage.createKeyValueStoreClient({ name: 'record-key-store' });
+ const client = await storage.createKeyValueStoreBackend({ name: 'record-key-store' });
await expect(
client.setValue({ key: '../escaped-record', value: 'pwned', contentType: 'text/plain' }),
).rejects.toThrow();
-
- // The escaped file must not have been created outside the store directory.
- await expect(pathExists(resolve(storage.keyValueStoresDirectory, 'escaped-record.txt'))).resolves.toBe(
- false,
- );
});
test('setValue still works for a regular key', async () => {
- const client = await storage.createKeyValueStoreClient({ name: 'record-key-store-ok' });
+ const client = await storage.createKeyValueStoreBackend({ name: 'record-key-store-ok' });
await expect(
client.setValue({ key: 'SAFEKEY', value: 'value', contentType: 'text/plain' }),
).resolves.not.toThrow();
- // The store was opened by name, so it lives under its name directory.
- const storePath = resolve(storage.keyValueStoresDirectory, (await client.getMetadata()).name!);
- await waitTillWrittenToDisk(resolve(storePath, 'SAFEKEY.txt'));
- await expect(pathExists(resolve(storePath, 'SAFEKEY.txt'))).resolves.toBe(true);
+ expect(await client.recordExists('SAFEKEY')).toBe(true);
});
});
diff --git a/packages/fs-storage/test/key-value-store/special-keys.test.ts b/packages/fs-storage/test/key-value-store/special-keys.test.ts
index 3d4311372..fe6c54c30 100644
--- a/packages/fs-storage/test/key-value-store/special-keys.test.ts
+++ b/packages/fs-storage/test/key-value-store/special-keys.test.ts
@@ -1,7 +1,7 @@
import { rm } from 'node:fs/promises';
import { resolve } from 'node:path';
-import { FileSystemStorageClient } from '@crawlee/fs-storage';
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
// Keys may contain characters that are unsafe in a file name (e.g. `.` or `/`). The adapter must
// round-trip such keys correctly through `setValue` / `getValue` / `listKeys` regardless of how the
@@ -19,8 +19,8 @@ describe('KeyValueStore handles keys with file-name-unsafe characters', () => {
});
test('round-trips a key containing a dot', async () => {
- const storage = new FileSystemStorageClient({ localDataDirectory: tmpLocation });
- const store = await storage.createKeyValueStoreClient({ name: 'dotted' });
+ const storage = new FileSystemStorageBackend({ localDataDirectory: tmpLocation });
+ const store = await storage.createKeyValueStoreBackend({ name: 'dotted' });
const body = '<html lang="en"><body>Hi there!</body></html>';
await store.setValue({
@@ -39,8 +39,8 @@ describe('KeyValueStore handles keys with file-name-unsafe characters', () => {
});
test('round-trips a key containing a slash', async () => {
- const storage = new FileSystemStorageClient({ localDataDirectory: tmpLocation });
- const store = await storage.createKeyValueStoreClient({ name: 'slashed' });
+ const storage = new FileSystemStorageBackend({ localDataDirectory: tmpLocation });
+ const store = await storage.createKeyValueStoreBackend({ name: 'slashed' });
const body = JSON.stringify({ ok: true });
await store.setValue({ key: 'nested/key', value: body, contentType: 'application/json; charset=utf-8' });
diff --git a/packages/fs-storage/test/key-value-store/stream.test.ts b/packages/fs-storage/test/key-value-store/stream.test.ts
index 85d2743c3..f3cddfc9f 100644
--- a/packages/fs-storage/test/key-value-store/stream.test.ts
+++ b/packages/fs-storage/test/key-value-store/stream.test.ts
@@ -2,11 +2,11 @@ import { rm } from 'node:fs/promises';
import { resolve } from 'node:path';
import { Readable } from 'node:stream';
-import { FileSystemStorageClient } from '@crawlee/fs-storage';
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
describe('KeyValueStore should drain streams when setting records', () => {
const localDataDirectory = resolve(__dirname, './tmp/stream');
- const storage = new FileSystemStorageClient({ localDataDirectory });
+ const storage = new FileSystemStorageBackend({ localDataDirectory });
const fsStream = Readable.from([Buffer.from('hello'), Buffer.from('world')]);
@@ -15,7 +15,7 @@ describe('KeyValueStore should drain streams when setting records', () => {
});
test('should drain stream', async () => {
- const defaultStore = await storage.createKeyValueStoreClient({ name: 'default' });
+ const defaultStore = await storage.createKeyValueStoreBackend({ name: 'default' });
await defaultStore.setValue({ key: 'streamz', value: fsStream, contentType: 'text/plain' });
diff --git a/packages/fs-storage/test/no-crash-on-big-buffers.test.ts b/packages/fs-storage/test/no-crash-on-big-buffers.test.ts
index aa1ad7d5a..d165c1d71 100644
--- a/packages/fs-storage/test/no-crash-on-big-buffers.test.ts
+++ b/packages/fs-storage/test/no-crash-on-big-buffers.test.ts
@@ -7,17 +7,17 @@
import { rm } from 'node:fs/promises';
import { resolve } from 'node:path';
-import { FileSystemStorageClient } from '@crawlee/fs-storage';
-import type { KeyValueStoreClient } from '@crawlee/types';
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
+import type { KeyValueStoreBackend } from '@crawlee/types';
describe('KeyValueStore round-trips a large binary value', () => {
const tmpLocation = resolve(import.meta.dirname, './tmp/no-buffer-crash');
- const storage = new FileSystemStorageClient({ localDataDirectory: tmpLocation });
+ const storage = new FileSystemStorageBackend({ localDataDirectory: tmpLocation });
- let store: KeyValueStoreClient;
+ let store: KeyValueStoreBackend;
beforeAll(async () => {
- store = await storage.createKeyValueStoreClient();
+ store = await storage.createKeyValueStoreBackend();
});
afterAll(async () => {
diff --git a/packages/fs-storage/test/request-queue/adapter.test.ts b/packages/fs-storage/test/request-queue/adapter.test.ts
index cea9641c0..425c2cc7f 100644
--- a/packages/fs-storage/test/request-queue/adapter.test.ts
+++ b/packages/fs-storage/test/request-queue/adapter.test.ts
@@ -1,24 +1,24 @@
import { rm } from 'node:fs/promises';
import { resolve } from 'node:path';
-import { FileSystemStorageClient } from '@crawlee/fs-storage';
-import type { RequestQueueClient } from '@crawlee/types';
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
+import type { RequestQueueBackend } from '@crawlee/types';
// The request-queue ordering, locking and finished-ness semantics are owned (and exhaustively tested)
// by the native `@crawlee/fs-storage-native` extension. These tests cover what the *adapter* adds on
// top: mapping requests and operation results between the native shapes and the `@crawlee/types`
// interfaces, and a thin lifecycle smoke test to catch wiring regressions.
-describe('RequestQueueClient adapter', () => {
+describe('RequestQueueBackend adapter', () => {
const tmpLocation = resolve(import.meta.dirname, './tmp/adapter');
- let requestQueue: RequestQueueClient;
+ let requestQueue: RequestQueueBackend;
let testIndex = 0;
beforeEach(async () => {
// Isolate each test with its own storage directory and queue so persisted counts/requests from
// one test cannot leak into the next.
- const storage = new FileSystemStorageClient({ localDataDirectory: resolve(tmpLocation, `${testIndex++}`) });
- requestQueue = await storage.createRequestQueueClient({ name: 'adapter' });
+ const storage = new FileSystemStorageBackend({ localDataDirectory: resolve(tmpLocation, `${testIndex++}`) });
+ requestQueue = await storage.createRequestQueueBackend({ name: 'adapter' });
});
afterAll(async () => {
diff --git a/packages/fs-storage/test/request-queue/reload-persistence.test.ts b/packages/fs-storage/test/request-queue/reload-persistence.test.ts
index 8714ed730..f71ecd77f 100644
--- a/packages/fs-storage/test/request-queue/reload-persistence.test.ts
+++ b/packages/fs-storage/test/request-queue/reload-persistence.test.ts
@@ -1,11 +1,11 @@
import { rm } from 'node:fs/promises';
import { resolve } from 'node:path';
-import { FileSystemStorageClient } from '@crawlee/fs-storage';
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
// The native client owns request-queue persistence; what this test exercises is the *adapter* wiring
-// that drives it: `FileSystemStorageClient.teardown()` must flush every opened queue's state via
-// `persistState()`, and reopening through a fresh `FileSystemStorageClient` over the same directory
+// that drives it: `FileSystemStorageBackend.teardown()` must flush every opened queue's state via
+// `persistState()`, and reopening through a fresh `FileSystemStorageBackend` over the same directory
// must restore the pending requests (with the adapter's request mapping intact).
describe('Request queue persists across reopen via teardown', () => {
const tmpLocation = resolve(import.meta.dirname, './tmp/req-queue-reload');
@@ -15,8 +15,8 @@ describe('Request queue persists across reopen via teardown', () => {
});
test('requests added and persisted are restored when the queue is reopened', async () => {
- const storage = new FileSystemStorageClient({ localDataDirectory: tmpLocation });
- const queue = await storage.createRequestQueueClient({ name: 'default' });
+ const storage = new FileSystemStorageBackend({ localDataDirectory: tmpLocation });
+ const queue = await storage.createRequestQueueBackend({ name: 'default' });
await queue.addBatchOfRequests([
{ url: 'http://example.com/1', uniqueKey: '1' },
@@ -27,8 +27,8 @@ describe('Request queue persists across reopen via teardown', () => {
await storage.teardown();
// Reopen over the same directory, emulating a fresh process.
- const reopenedStorage = new FileSystemStorageClient({ localDataDirectory: tmpLocation });
- const reopenedQueue = await reopenedStorage.createRequestQueueClient({ name: 'default' });
+ const reopenedStorage = new FileSystemStorageBackend({ localDataDirectory: tmpLocation });
+ const reopenedQueue = await reopenedStorage.createRequestQueueBackend({ name: 'default' });
const metadata = await reopenedQueue.getMetadata();
expect(metadata.pendingRequestCount).toBe(2);
diff --git a/packages/fs-storage/test/request-queue/request-queue-access.test.ts b/packages/fs-storage/test/request-queue/request-queue-access.test.ts
index ec05d3450..f3b8bc1c3 100644
--- a/packages/fs-storage/test/request-queue/request-queue-access.test.ts
+++ b/packages/fs-storage/test/request-queue/request-queue-access.test.ts
@@ -1,14 +1,14 @@
import { rm } from 'node:fs/promises';
import { resolve } from 'node:path';
-import { FileSystemStorageClient } from '@crawlee/fs-storage';
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
// `requestQueueAccess` controls how the native `@crawlee/fs-storage-native` extension treats requests
// left *in progress* by a previous run (a dangling `orderNo` lock on disk) when a queue is reopened.
// The reclaim/respect-peer-lock semantics are owned by the native extension; these tests verify the
// adapter's contract on top of it: the option defaults to `'single'`, is honored when set to
// `'shared'`, and that the resulting behavior reaches all the way down to the native queue.
-describe('FileSystemStorageClient requestQueueAccess', () => {
+describe('FileSystemStorageBackend requestQueueAccess', () => {
const tmpLocation = resolve(import.meta.dirname, './tmp/request-queue-access');
afterEach(async () => {
@@ -16,20 +16,20 @@ describe('FileSystemStorageClient requestQueueAccess', () => {
});
test("defaults to 'single'", () => {
- const storage = new FileSystemStorageClient({ localDataDirectory: tmpLocation });
+ const storage = new FileSystemStorageBackend({ localDataDirectory: tmpLocation });
expect(storage.requestQueueAccess).toBe('single');
});
test("respects an explicit 'shared'", () => {
- const storage = new FileSystemStorageClient({ localDataDirectory: tmpLocation, requestQueueAccess: 'shared' });
+ const storage = new FileSystemStorageBackend({ localDataDirectory: tmpLocation, requestQueueAccess: 'shared' });
expect(storage.requestQueueAccess).toBe('shared');
});
// Seed a queue with two requests, fetch (lock) one without handling it or tearing down — leaving a
// dangling in-progress lock on disk, exactly the "process died mid-flight" situation.
async function seedQueueWithDanglingLock(dir: string) {
- const storage = new FileSystemStorageClient({ localDataDirectory: dir });
- const queue = await storage.createRequestQueueClient({ name: 'default' });
+ const storage = new FileSystemStorageBackend({ localDataDirectory: dir });
+ const queue = await storage.createRequestQueueBackend({ name: 'default' });
await queue.addBatchOfRequests([
{ url: 'http://example.com/1', uniqueKey: '1' },
{ url: 'http://example.com/2', uniqueKey: '2' },
@@ -45,8 +45,8 @@ describe('FileSystemStorageClient requestQueueAccess', () => {
const locked = await seedQueueWithDanglingLock(dir);
// Reopen the same directory as sole owner, without purging.
- const reopened = new FileSystemStorageClient({ localDataDirectory: dir, requestQueueAccess: 'single' });
- const queue = await reopened.createRequestQueueClient({ name: 'default' });
+ const reopened = new FileSystemStorageBackend({ localDataDirectory: dir, requestQueueAccess: 'single' });
+ const queue = await reopened.createRequestQueueBackend({ name: 'default' });
// Contents preserved: both requests still present, none handled.
const metadata = await queue.getMetadata();
@@ -69,8 +69,8 @@ describe('FileSystemStorageClient requestQueueAccess', () => {
// Reopen in concurrency-safe mode: an in-progress request is treated as a potential live peer's
// lock and is NOT reclaimed until it expires.
- const reopened = new FileSystemStorageClient({ localDataDirectory: dir, requestQueueAccess: 'shared' });
- const queue = await reopened.createRequestQueueClient({ name: 'default' });
+ const reopened = new FileSystemStorageBackend({ localDataDirectory: dir, requestQueueAccess: 'shared' });
+ const queue = await reopened.createRequestQueueBackend({ name: 'default' });
// Contents are still preserved...
const metadata = await queue.getMetadata();
diff --git a/packages/fs-storage/test/reverse-datataset-list.test.ts b/packages/fs-storage/test/reverse-datataset-list.test.ts
index dbafc72ab..c6b3e43c9 100644
--- a/packages/fs-storage/test/reverse-datataset-list.test.ts
+++ b/packages/fs-storage/test/reverse-datataset-list.test.ts
@@ -1,25 +1,25 @@
import { rm } from 'node:fs/promises';
import { resolve } from 'node:path';
-import { FileSystemStorageClient } from '@crawlee/fs-storage';
-import type { DatasetClient } from '@crawlee/types';
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
+import type { DatasetBackend } from '@crawlee/types';
const elements = Array.from({ length: 10 }, (_, i) => ({ number: i }));
describe('Dataset#getData respects the desc option', () => {
const localDataDirectory = resolve(import.meta.dirname, './tmp/desc');
- const storage = new FileSystemStorageClient({
+ const storage = new FileSystemStorageBackend({
localDataDirectory,
});
- let dataset: DatasetClient;
+ let dataset: DatasetBackend;
afterAll(async () => {
await rm(localDataDirectory, { force: true, recursive: true });
});
beforeAll(async () => {
- dataset = await storage.createDatasetClient({ name: 'false' });
+ dataset = await storage.createDatasetBackend({ name: 'false' });
await dataset.pushData(elements);
});
diff --git a/packages/fs-storage/test/storage-name-path-traversal.test.ts b/packages/fs-storage/test/storage-name-path-traversal.test.ts
index 00f5aa14d..d036b97d8 100644
--- a/packages/fs-storage/test/storage-name-path-traversal.test.ts
+++ b/packages/fs-storage/test/storage-name-path-traversal.test.ts
@@ -1,7 +1,7 @@
import { rm } from 'node:fs/promises';
import { resolve, sep } from 'node:path';
-import { FileSystemStorageClient } from '@crawlee/fs-storage';
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
describe('storage name path traversal', () => {
const tmpLocation = resolve(import.meta.dirname, './tmp/storage-name-path-traversal');
@@ -10,29 +10,28 @@ describe('storage name path traversal', () => {
await rm(tmpLocation, { force: true, recursive: true });
});
- const storage = new FileSystemStorageClient({
+ const storage = new FileSystemStorageBackend({
localDataDirectory: tmpLocation,
- writeMetadata: true,
});
const traversalNames = ['../escaped', `..${sep}escaped`, resolve(tmpLocation, '..', 'escaped-absolute')];
- describe('createXClient rejects names that escape the storage directory', () => {
+ describe('createXBackend rejects names that escape the storage directory', () => {
test.each(traversalNames)('key-value store name %s', async (name) => {
- await expect(storage.createKeyValueStoreClient({ name })).rejects.toThrow();
+ await expect(storage.createKeyValueStoreBackend({ name })).rejects.toThrow();
});
test.each(traversalNames)('dataset name %s', async (name) => {
- await expect(storage.createDatasetClient({ name })).rejects.toThrow();
+ await expect(storage.createDatasetBackend({ name })).rejects.toThrow();
});
test.each(traversalNames)('request queue name %s', async (name) => {
- await expect(storage.createRequestQueueClient({ name })).rejects.toThrow();
+ await expect(storage.createRequestQueueBackend({ name })).rejects.toThrow();
});
});
test('legitimate names still work', async () => {
- const client = await storage.createKeyValueStoreClient({ name: 'normal-name' });
+ const client = await storage.createKeyValueStoreBackend({ name: 'normal-name' });
await expect(
client.setValue({ key: 'SAFEKEY', value: 'value', contentType: 'text/plain' }),
diff --git a/packages/types/src/storages.ts b/packages/types/src/storages.ts
index 772504769..084e2902f 100644
--- a/packages/types/src/storages.ts
+++ b/packages/types/src/storages.ts
@@ -16,7 +16,7 @@ export interface QueueOperationInfo {
}
/**
- * A single page of items returned by {@link DatasetClient.getData}.
+ * A single page of items returned by {@link DatasetBackend.getData}.
*
* Datasets paginate by offset, so a page is self-describing via `total` / `offset` / `limit`: a
* frontend assembling all pages knows it has reached the end once `offset + items.length >= total`.
@@ -37,7 +37,7 @@ export interface PaginatedList<Data> {
items: Data[];
}
-export interface DatasetClientListOptions {
+export interface DatasetBackendListOptions {
desc?: boolean;
limit?: number;
offset?: number;
@@ -52,7 +52,7 @@ export interface DatasetInfo {
itemCount: number;
}
-export interface DatasetClient<Data extends Dictionary = Dictionary> {
+export interface DatasetBackend<Data extends Dictionary = Dictionary> {
/**
* Returns metadata about the dataset (id, name, timestamps, item count, etc.).
*
@@ -72,7 +72,7 @@ export interface DatasetClient<Data extends Dictionary = Dictionary> {
pushData(items: Data[]): Promise<void>;
/** Fetch a page of items from the dataset. */
- getData(options?: DatasetClientListOptions): Promise<PaginatedList<Data>>;
+ getData(options?: DatasetBackendListOptions): Promise<PaginatedList<Data>>;
}
export interface KeyValueStoreInfo {
@@ -84,7 +84,7 @@ export interface KeyValueStoreInfo {
}
/**
- * The value a serialized record carries on the way *into* a storage client (`setValue`).
+ * The value a serialized record carries on the way *into* a storage backend (`setValue`).
*
* The `KeyValueStore` frontend has already serialized by this point, so it is a pre-serialized
* string, raw bytes (`Buffer` / `ArrayBuffer` / typed array), or a stream the client drains.
@@ -98,9 +98,9 @@ export type KeyValueStoreRecordInputValue =
| ReadableStream;
/**
- * A record as returned by a storage client (`getValue`).
+ * A record as returned by a storage backend (`getValue`).
*
- * Storage clients are byte transports: they persist and return bytes verbatim and never serialize
+ * Storage backends are byte transports: they persist and return bytes verbatim and never serialize
* or parse. Interpretation is the `KeyValueStore` frontend's job (it parses according to the content
* type via `parseValue`). The value is therefore always raw bytes — a `Buffer` (Node) or an
* `ArrayBuffer` (browser backends).
@@ -112,7 +112,7 @@ export interface KeyValueStoreRecord {
}
/**
- * A record passed to a storage client for writing (`setValue`). Like {@link KeyValueStoreRecord} but
+ * A record passed to a storage backend for writing (`setValue`). Like {@link KeyValueStoreRecord} but
* with the lenient, pre-serialized {@link KeyValueStoreRecordInputValue} for the value.
*/
export interface KeyValueStoreInputRecord {
@@ -138,10 +138,10 @@ export interface KeyValueStoreItemData {
}
/**
- * A single page of keys returned by {@link KeyValueStoreClient.listKeys}.
+ * A single page of keys returned by {@link KeyValueStoreBackend.listKeys}.
*
- * This mirrors {@link PaginatedList} (the shape returned by {@link DatasetClient.getData}) so that
- * both listing operations on the storage-client layer return a self-describing page. The difference
+ * This mirrors {@link PaginatedList} (the shape returned by {@link DatasetBackend.getData}) so that
+ * both listing operations on the storage backend layer return a self-describing page. The difference
* is the pagination model: datasets are offset-based (`total` / `offset`), whereas key-value stores
* are cursor-based. A frontend assembling all pages should therefore not guess "is this the last
* page?" from `items.length < limit` — it should rely on {@link isTruncated} and resume from
@@ -165,7 +165,7 @@ export interface KeyValueStoreListKeysResult {
/**
* Key-value Store client.
*/
-export interface KeyValueStoreClient {
+export interface KeyValueStoreBackend {
/**
* Returns metadata about the key-value store (id, name, timestamps, etc.).
*
@@ -199,7 +199,7 @@ export interface KeyValueStoreClient {
* List a single page of keys in the store. Returns at most `limit` keys starting after
* `exclusiveStartKey`, wrapped in a self-describing page.
*
- * Like {@link DatasetClient.getData}, this returns one page rather than the whole collection;
+ * Like {@link DatasetBackend.getData}, this returns one page rather than the whole collection;
* assembling all pages (e.g. for `KeyValueStore.keys()`) is the frontend's job. The result carries
* a cursor (`isTruncated` / `nextExclusiveStartKey`) so the frontend can paginate deterministically
* instead of inferring the end from `items.length < limit`.
@@ -226,7 +226,7 @@ export interface RequestQueueInfo {
/**
* Options for request-queue operations that add or return requests to the queue
- * ({@link RequestQueueClient.addBatchOfRequests}, {@link RequestQueueClient.reclaimRequest}).
+ * ({@link RequestQueueBackend.addBatchOfRequests}, {@link RequestQueueBackend.reclaimRequest}).
*/
export interface RequestQueueOperationOptions {
/** Place the affected request(s) at the beginning of the queue so they are processed sooner. */
@@ -278,7 +278,7 @@ export interface BatchAddRequestsResult {
* locking on the Apify platform) is an internal concern of the implementation and is not exposed on
* this interface.
*/
-export interface RequestQueueClient {
+export interface RequestQueueBackend {
/**
* Returns metadata about the request queue (id, name, timestamps, request counts, etc.).
*
@@ -405,19 +405,19 @@ export type StorageIdentifier =
| { id?: never; name?: never; alias?: never };
/**
- * Options for creating a dataset client via {@apilink StorageClient.createDatasetClient}.
+ * Options for creating a dataset backend via {@apilink StorageBackend.createDatasetBackend}.
*/
-export type CreateDatasetClientOptions = StorageIdentifier;
+export type CreateDatasetBackendOptions = StorageIdentifier;
/**
- * Options for creating a key-value store client via {@apilink StorageClient.createKeyValueStoreClient}.
+ * Options for creating a key-value store backend via {@apilink StorageBackend.createKeyValueStoreBackend}.
*/
-export type CreateKeyValueStoreClientOptions = StorageIdentifier;
+export type CreateKeyValueStoreBackendOptions = StorageIdentifier;
/**
- * Options for creating a request queue client via {@apilink StorageClient.createRequestQueueClient}.
+ * Options for creating a request queue backend via {@apilink StorageBackend.createRequestQueueBackend}.
*/
-export type CreateRequestQueueClientOptions = StorageIdentifier & {
+export type CreateRequestQueueBackendOptions = StorageIdentifier & {
/**
* Client key for request locking.
* TODO: This is an Apify-platform concern and should eventually be pushed down
@@ -438,36 +438,36 @@ export type CreateRequestQueueClientOptions = StorageIdentifier & {
* Represents a storage backend capable of working with datasets, key-value stores and request queues.
*
* A new storage backend needs to implement 4 classes:
- * - `StorageClient` - the factory that creates sub-clients
- * - `DatasetClient` - operations on a single dataset
- * - `KeyValueStoreClient` - operations on a single key-value store
- * - `RequestQueueClient` - operations on a single request queue
+ * - `StorageBackend` - the factory that creates sub-backends
+ * - `DatasetBackend` - operations on a single dataset
+ * - `KeyValueStoreBackend` - operations on a single key-value store
+ * - `RequestQueueBackend` - operations on a single request queue
*
- * The `StorageClient` acts as an async factory: each `create*` method either opens an existing
- * storage or creates a new one, returning a sub-client bound to that storage instance.
+ * The `StorageBackend` acts as an async factory: each `create*` method either opens an existing
+ * storage or creates a new one, returning a sub-backend bound to that storage instance.
*/
-export interface StorageClient {
+export interface StorageBackend {
/**
- * Create (or open) a dataset client.
+ * Create (or open) a dataset backend.
* If `id` is provided, opens the dataset with that ID.
* If `name` is provided, opens an existing dataset with that name or creates a new one.
* If neither is provided, opens or creates the default dataset.
*/
- createDatasetClient(options?: CreateDatasetClientOptions): Promise<DatasetClient>;
+ createDatasetBackend(options?: CreateDatasetBackendOptions): Promise<DatasetBackend>;
/**
- * Create (or open) a key-value store client.
+ * Create (or open) a key-value store backend.
* If `id` is provided, opens the key-value store with that ID.
* If `name` is provided, opens an existing store with that name or creates a new one.
* If neither is provided, opens or creates the default key-value store.
*/
- createKeyValueStoreClient(options?: CreateKeyValueStoreClientOptions): Promise<KeyValueStoreClient>;
+ createKeyValueStoreBackend(options?: CreateKeyValueStoreBackendOptions): Promise<KeyValueStoreBackend>;
/**
- * Create (or open) a request queue client.
+ * Create (or open) a request queue backend.
* If `id` is provided, opens the request queue with that ID.
* If `name` is provided, opens an existing queue with that name or creates a new one.
* If neither is provided, opens or creates the default request queue.
*/
- createRequestQueueClient(options?: CreateRequestQueueClientOptions): Promise<RequestQueueClient>;
+ createRequestQueueBackend(options?: CreateRequestQueueBackendOptions): Promise<RequestQueueBackend>;
/**
* Check whether a storage with the given ID exists.
*
@@ -480,12 +480,12 @@ export interface StorageClient {
*
* The key is used by `StorageInstanceManager` to partition the storage cache per-backend,
* so that two storages with the same name but backed by different clients
- * (e.g. a local `MemoryStorageClient` and a cloud `ApifyClient`) are cached as separate instances.
+ * (e.g. a local `MemoryStorageBackend` and a cloud `ApifyClient`) are cached as separate instances.
*
* When not provided, the fallback uses the client's constructor name, so different
- * `StorageClient` implementations automatically get separate cache partitions.
+ * `StorageBackend` implementations automatically get separate cache partitions.
*/
- getStorageClientCacheKey?(): string;
+ getStorageBackendCacheKey?(): string;
purge?(): Promise<void>;
teardown?(): Promise<void>;
stats?: { rateLimitErrors: number[] };
diff --git a/packages/utils/src/internals/general.ts b/packages/utils/src/internals/general.ts
index cb623691c..3158e8899 100644
--- a/packages/utils/src/internals/general.ts
+++ b/packages/utils/src/internals/general.ts
@@ -205,7 +205,7 @@ export function isStream(value: unknown): value is NodeJS.ReadableStream | Reada
// A Node.js Readable is both pipeable and async-iterable; a Web ReadableStream exposes pipeTo.
// Requiring async-iterability for the `pipe` branch rejects plain `{ pipe }` ducks that would
- // otherwise blow up later in the storage clients' drain loop with a cryptic TypeError.
+ // otherwise blow up later in the storage backends' drain loop with a cryptic TypeError.
const isNodeStream =
typeof (value as any).pipe === 'function' && typeof (value as any)[Symbol.asyncIterator] === 'function';
const isWebStream = typeof (value as any).pipeTo === 'function';
@@ -231,7 +231,7 @@ export function isBuffer(value: unknown): value is Buffer | ArrayBuffer | ArrayB
/**
* Converts a byte-like value (Buffer, ArrayBuffer, or any typed-array / DataView) into a Buffer over
* the exact same bytes, honoring `byteOffset` / `byteLength` for views. Existing Buffers are returned
- * as-is. Used by storage clients, which persist raw bytes regardless of the input's concrete shape.
+ * as-is. Used by storage backends, which persist raw bytes regardless of the input's concrete shape.
* @ignore
*/
export function toBuffer(value: Buffer | ArrayBuffer | ArrayBufferView): Buffer {
diff --git a/test/core/autoscaling/snapshotter.test.ts b/test/core/autoscaling/snapshotter.test.ts
index 7d133a138..173cf7794 100644
--- a/test/core/autoscaling/snapshotter.test.ts
+++ b/test/core/autoscaling/snapshotter.test.ts
@@ -25,7 +25,7 @@ describe('Snapshotter', () => {
serviceLocator.setConfiguration(new Configuration({ systemInfoIntervalMillis: 100 }));
// mock client data
- const apifyClient = serviceLocator.getStorageClient();
+ const apifyClient = serviceLocator.getStorageBackend();
const oldStats = apifyClient.stats;
apifyClient.stats = {} as any;
apifyClient.stats!.rateLimitErrors = [0, 0, 0];
@@ -279,7 +279,7 @@ describe('Snapshotter', () => {
test('correctly marks clientOverloaded', () => {
// mock client data
- const apifyClient = serviceLocator.getStorageClient();
+ const apifyClient = serviceLocator.getStorageBackend();
const oldStats = apifyClient.stats;
apifyClient.stats = {} as any;
apifyClient.stats!.rateLimitErrors = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
diff --git a/test/core/crawlers/adaptive_playwright_crawler.test.ts b/test/core/crawlers/adaptive_playwright_crawler.test.ts
index 6f6c14d68..6696c628b 100644
--- a/test/core/crawlers/adaptive_playwright_crawler.test.ts
+++ b/test/core/crawlers/adaptive_playwright_crawler.test.ts
@@ -9,7 +9,7 @@ import {
type Dictionary,
EventType,
KeyValueStore,
- MemoryStorageClient,
+ MemoryStorageBackend,
serviceLocator,
} from '@crawlee/core';
import type {
@@ -132,8 +132,8 @@ describe('AdaptivePlaywrightCrawler', () => {
beforeEach(async () => {
// The global test setup (`test/vitest.setup.ts`) already calls `serviceLocator.reset()` before
// each test, which clears the storage-instance cache; here we just install a fresh in-memory
- // storage client for this suite.
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ // storage backend for this suite.
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
// `BasicCrawler` keeps a process-global instance counter that assigns each crawler a distinct
// default request queue (the first one uses the shared default queue, later ones get their own
// `__default_<n>__` alias). Since every test wipes storage and starts fresh, the counter must be
diff --git a/test/core/crawlers/basic_crawler.test.ts b/test/core/crawlers/basic_crawler.test.ts
index fc5212849..e0d9d4be9 100644
--- a/test/core/crawlers/basic_crawler.test.ts
+++ b/test/core/crawlers/basic_crawler.test.ts
@@ -22,7 +22,7 @@ import {
serviceLocator,
SessionPool,
} from '@crawlee/basic';
-import { Dataset, MemoryStorageClient, RequestState, SessionPool, Statistics } from '@crawlee/core';
+import { Dataset, MemoryStorageBackend, RequestState, SessionPool, Statistics } from '@crawlee/core';
import type { ISession, ProxyInfo } from '@crawlee/types';
import type { Dictionary } from '@crawlee/utils';
import { RobotsTxtFile, sleep } from '@crawlee/utils';
@@ -36,11 +36,11 @@ import { startExpressAppPromise } from 'test/shared/_helper.js';
import log from '@apify/log';
-type MemoryRequestQueueClient = Awaited<ReturnType<MemoryStorageClient['createRequestQueueClient']>>;
+type MemoryRequestQueueBackend = Awaited<ReturnType<MemoryStorageBackend['createRequestQueueBackend']>>;
describe('BasicCrawler', () => {
let logLevel: number;
- let requestQueueClient: MemoryRequestQueueClient;
+ let requestQueueBackend: MemoryRequestQueueBackend;
const HOSTNAME = '127.0.0.1';
let port: number;
@@ -64,9 +64,9 @@ describe('BasicCrawler', () => {
beforeEach(async () => {
vitest.clearAllMocks();
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
const memoryRequestQueue = await RequestQueue.open();
- requestQueueClient = memoryRequestQueue.client as MemoryRequestQueueClient;
+ requestQueueBackend = memoryRequestQueue.backend as MemoryRequestQueueBackend;
});
afterAll(async () => {
@@ -202,14 +202,14 @@ describe('BasicCrawler', () => {
expect(processed).toHaveLength(2);
// Make sure no extra requests were enqueued
- await expect(requestQueueClient.listItems()).resolves.toEqual([]);
+ await expect(requestQueueBackend.listItems()).resolves.toEqual([]);
// Second run should process 2 more requests
await crawler.run([...Array(5).keys()].map((index) => `https://example.com/second/${index}`));
expect(processed).toHaveLength(4);
// Make sure no extra requests were enqueued
- await expect(requestQueueClient.listItems()).resolves.toEqual([]);
+ await expect(requestQueueBackend.listItems()).resolves.toEqual([]);
const processedUrls = processed.map((p) => p.url);
@@ -1893,7 +1893,7 @@ describe('BasicCrawler', () => {
// Should only have added the first 3 requests (since 2 were already processed, limit allows 3 more)
expect(addRequestsBatchedSpy).toHaveBeenCalledOnce();
- await expect(requestQueueClient.listItems()).resolves.toMatchObject([
+ await expect(requestQueueBackend.listItems()).resolves.toMatchObject([
{ url: 'http://example.com/1' },
{ url: 'http://example.com/2' },
{ url: 'http://example.com/3' },
@@ -1915,7 +1915,7 @@ describe('BasicCrawler', () => {
// First call - should add 2 requests (2 more slots to go)
await crawler.addRequests(['http://example.com/1', 'http://example.com/2']);
- await expect(requestQueueClient.listItems()).resolves.toMatchObject([
+ await expect(requestQueueBackend.listItems()).resolves.toMatchObject([
{ url: 'http://example.com/1' },
{ url: 'http://example.com/2' },
]);
@@ -1928,7 +1928,7 @@ describe('BasicCrawler', () => {
'http://example.com/6', // This should be ignored
]);
- await expect(requestQueueClient.listItems()).resolves.toMatchObject([
+ await expect(requestQueueBackend.listItems()).resolves.toMatchObject([
{ url: 'http://example.com/1' },
{ url: 'http://example.com/2' },
{ url: 'http://example.com/3' },
@@ -1938,7 +1938,7 @@ describe('BasicCrawler', () => {
// Third call - should add no requests (limit already reached)
await crawler.addRequests(['http://example.com/7', 'http://example.com/8']);
- await expect(requestQueueClient.listItems()).resolves.toMatchObject([
+ await expect(requestQueueBackend.listItems()).resolves.toMatchObject([
{ url: 'http://example.com/1' },
{ url: 'http://example.com/2' },
{ url: 'http://example.com/3' },
@@ -1973,7 +1973,7 @@ describe('BasicCrawler', () => {
'http://example.com/4', // Would exceed limit
]);
- await expect(requestQueueClient.listItems()).resolves.toMatchObject([
+ await expect(requestQueueBackend.listItems()).resolves.toMatchObject([
{ url: 'http://example.com/1' },
{ url: 'http://example.com/3' },
]);
@@ -2035,7 +2035,7 @@ describe('BasicCrawler', () => {
'http://example.com/my-crawler/anything', // Blocked by robots.txt for all user-agents, but allowed for "MyCrawler"
]);
- await expect(requestQueueClient.listItems()).resolves.toMatchObject(visitedUrls);
+ await expect(requestQueueBackend.listItems()).resolves.toMatchObject(visitedUrls);
// Should only have added the first request (allowed by robots.txt and within limit)
expect(addRequestsBatchedSpy).toHaveBeenCalledOnce();
@@ -2300,7 +2300,7 @@ describe('BasicCrawler', () => {
await crawler.addRequests(requestsToAdd);
// Both unique URLs should have been enqueued — duplicates should not consume the budget
- await expect(requestQueueClient.listItems()).resolves.toMatchObject([
+ await expect(requestQueueBackend.listItems()).resolves.toMatchObject([
{ url: 'http://example.com/same' },
{ url: 'http://example.com/new' },
]);
@@ -2695,14 +2695,14 @@ describe('BasicCrawler', () => {
await rm(`${tmpDir}/result.csv`);
});
- test("Crawlers with different storage clients don't share Datasets", async () => {
- // Each crawler gets its own MemoryStorageClient instance; every instance has a unique
+ test("Crawlers with different storage backends don't share Datasets", async () => {
+ // Each crawler gets its own MemoryStorageBackend instance; every instance has a unique
// per-instance cache key, so they end up in separate cache partitions.
- const storageA = new MemoryStorageClient();
- const storageB = new MemoryStorageClient();
+ const storageA = new MemoryStorageBackend();
+ const storageB = new MemoryStorageBackend();
- const crawlerA = new BasicCrawler({ storageClient: storageA });
- const crawlerB = new BasicCrawler({ storageClient: storageB });
+ const crawlerA = new BasicCrawler({ storageBackend: storageA });
+ const crawlerB = new BasicCrawler({ storageBackend: storageB });
await crawlerA.pushData(getPayload('A'));
await crawlerB.pushData(getPayload('B'));
@@ -2712,17 +2712,17 @@ describe('BasicCrawler', () => {
expect((await crawlerB.getData()).items).toEqual(getPayload('B'));
});
- test('Crawlers with different storage clients run separately', async () => {
- const storageA = new MemoryStorageClient();
- const storageB = new MemoryStorageClient();
+ test('Crawlers with different storage backends run separately', async () => {
+ const storageA = new MemoryStorageBackend();
+ const storageB = new MemoryStorageBackend();
const crawlerA = new BasicCrawler({
requestHandler: () => {},
- storageClient: storageA,
+ storageBackend: storageA,
});
const crawlerB = new BasicCrawler({
requestHandler: () => {},
- storageClient: storageB,
+ storageBackend: storageB,
});
await crawlerA.run([{ url: `http://${HOSTNAME}:${port}` }]);
diff --git a/test/core/crawlers/browser_crawler.test.ts b/test/core/crawlers/browser_crawler.test.ts
index 217d27a10..7d9c41d4e 100644
--- a/test/core/crawlers/browser_crawler.test.ts
+++ b/test/core/crawlers/browser_crawler.test.ts
@@ -10,7 +10,7 @@ import {
import {
bindMethodsToServiceLocator,
BLOCKED_STATUS_CODES,
- MemoryStorageClient,
+ MemoryStorageBackend,
ServiceLocator,
SessionPool,
} from '@crawlee/core';
@@ -56,7 +56,7 @@ describe('BrowserCrawler', () => {
aroundEach(async (t) => {
const scopedServiceLocator = new ServiceLocator();
- scopedServiceLocator.setStorageClient(new MemoryStorageClient());
+ scopedServiceLocator.setStorageBackend(new MemoryStorageBackend());
const { run } = bindMethodsToServiceLocator(scopedServiceLocator, {});
await run(t);
diff --git a/test/core/crawlers/cheerio_crawler.test.ts b/test/core/crawlers/cheerio_crawler.test.ts
index 63f69dbb0..7e5756457 100644
--- a/test/core/crawlers/cheerio_crawler.test.ts
+++ b/test/core/crawlers/cheerio_crawler.test.ts
@@ -11,7 +11,7 @@ import {
RequestList,
Session,
} from '@crawlee/cheerio';
-import { BaseCrawleeLogger, MemoryStorageClient, serviceLocator, SessionPool } from '@crawlee/core';
+import { BaseCrawleeLogger, MemoryStorageBackend, serviceLocator, SessionPool } from '@crawlee/core';
import { ImpitHttpClient } from '@crawlee/impit-client';
import type { ISession, ProxyInfo } from '@crawlee/types';
import type { Dictionary } from '@crawlee/utils';
@@ -72,7 +72,7 @@ describe('CheerioCrawler', () => {
});
beforeEach(async () => {
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
});
afterAll(async () => {
diff --git a/test/core/crawlers/dom_crawler.test.ts b/test/core/crawlers/dom_crawler.test.ts
index 86d97aa65..458d09cd3 100644
--- a/test/core/crawlers/dom_crawler.test.ts
+++ b/test/core/crawlers/dom_crawler.test.ts
@@ -1,7 +1,7 @@
import http from 'node:http';
import type { AddressInfo } from 'node:net';
-import { MemoryStorageClient, serviceLocator } from '@crawlee/core';
+import { MemoryStorageBackend, serviceLocator } from '@crawlee/core';
import { JSDOMCrawler } from '@crawlee/jsdom';
import { LinkeDOMCrawler } from '@crawlee/linkedom';
@@ -46,7 +46,7 @@ afterAll(async () => {
});
beforeEach(async () => {
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
});
test('works', async () => {
diff --git a/test/core/crawlers/http_crawler.test.ts b/test/core/crawlers/http_crawler.test.ts
index 1a0f288d5..cba6ab62b 100644
--- a/test/core/crawlers/http_crawler.test.ts
+++ b/test/core/crawlers/http_crawler.test.ts
@@ -2,7 +2,7 @@ import http from 'node:http';
import type { AddressInfo } from 'node:net';
import { Readable } from 'node:stream';
-import { MemoryStorageClient, serviceLocator } from '@crawlee/core';
+import { MemoryStorageBackend, serviceLocator } from '@crawlee/core';
import { HttpCrawler, SessionPool } from '@crawlee/http';
import { ResponseWithUrl } from '@crawlee/http-client';
import iconv from 'iconv-lite';
@@ -107,7 +107,7 @@ afterAll(async () => {
});
beforeEach(async () => {
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
});
test('works', async () => {
diff --git a/test/core/crawlers/playwright_crawler.test.ts b/test/core/crawlers/playwright_crawler.test.ts
index fd84da743..d4d1c2a0c 100644
--- a/test/core/crawlers/playwright_crawler.test.ts
+++ b/test/core/crawlers/playwright_crawler.test.ts
@@ -8,7 +8,7 @@ import type {
PlaywrightRequestHandler,
Request,
} from '@crawlee/playwright';
-import { MemoryStorageClient, serviceLocator } from '@crawlee/core';
+import { MemoryStorageBackend, serviceLocator } from '@crawlee/core';
import { createPlaywrightRouter, PlaywrightCrawler, RequestList, RequestValidationError } from '@crawlee/playwright';
import type { Cheerio, CheerioAPI, CheerioRoot, Element } from '@crawlee/utils';
import express from 'express';
@@ -58,7 +58,7 @@ describe('PlaywrightCrawler', () => {
});
beforeEach(async () => {
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
const sources = [`http://${HOSTNAME}:${[port]}/`];
requestList = await RequestList.open(`sources-${Math.random() * 10000}`, sources);
diff --git a/test/core/crawlers/puppeteer_crawler.test.ts b/test/core/crawlers/puppeteer_crawler.test.ts
index 6b62d1d02..e6b971bfa 100644
--- a/test/core/crawlers/puppeteer_crawler.test.ts
+++ b/test/core/crawlers/puppeteer_crawler.test.ts
@@ -24,7 +24,7 @@ import {
Session,
SessionPool,
} from '@crawlee/puppeteer';
-import { MemoryStorageClient, serviceLocator } from '@crawlee/core';
+import { MemoryStorageBackend, serviceLocator } from '@crawlee/core';
import { sleep } from '@crawlee/utils';
import type { Server as ProxyChainServer } from 'proxy-chain';
import { z } from 'zod';
@@ -76,7 +76,7 @@ describe('PuppeteerCrawler', () => {
});
beforeEach(async () => {
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
const sources = [serverUrl];
requestList = await RequestList.open(`sources-${Math.random() * 10000}`, sources);
diff --git a/test/core/crawlers/rendering_type_predictor.test.ts b/test/core/crawlers/rendering_type_predictor.test.ts
index 73152f178..e95545314 100644
--- a/test/core/crawlers/rendering_type_predictor.test.ts
+++ b/test/core/crawlers/rendering_type_predictor.test.ts
@@ -1,10 +1,10 @@
-import { KeyValueStore, MemoryStorageClient, Request, serviceLocator } from '@crawlee/core';
+import { KeyValueStore, MemoryStorageBackend, Request, serviceLocator } from '@crawlee/core';
import { RenderingTypePredictor } from '@crawlee/playwright';
import { beforeEach, describe, expect, it } from 'vitest';
describe('RenderingTypePredictor', () => {
beforeEach(async () => {
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
});
describe('persistence', () => {
diff --git a/test/core/crawlers/statistics.test.ts b/test/core/crawlers/statistics.test.ts
index 7b27d79fa..10354c637 100644
--- a/test/core/crawlers/statistics.test.ts
+++ b/test/core/crawlers/statistics.test.ts
@@ -1,4 +1,4 @@
-import { Configuration, EventType, MemoryStorageClient, serviceLocator, Statistics } from '@crawlee/core';
+import { Configuration, EventType, MemoryStorageBackend, serviceLocator, Statistics } from '@crawlee/core';
import type { Dictionary } from '@crawlee/utils';
describe('Statistics', () => {
@@ -12,7 +12,7 @@ describe('Statistics', () => {
});
beforeEach(async () => {
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
stats = new Statistics();
});
diff --git a/test/core/enqueue_links/click_elements.test.ts b/test/core/enqueue_links/click_elements.test.ts
index 5ab39711b..6b4d6a952 100644
--- a/test/core/enqueue_links/click_elements.test.ts
+++ b/test/core/enqueue_links/click_elements.test.ts
@@ -4,7 +4,7 @@ import type { RequestQueueOperationOptions, Source } from 'crawlee';
import {
launchPlaywright,
launchPuppeteer,
- MemoryStorageClient,
+ MemoryStorageBackend,
playwrightClickElements,
playwrightUtils,
puppeteerClickElements,
@@ -79,7 +79,7 @@ testCases.forEach(({ caseName, launchBrowser, clickElements, utils }) => {
beforeEach(async () => {
page = await browser.newPage();
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
});
afterEach(async () => {
diff --git a/test/core/enqueue_links/enqueue_links.test.ts b/test/core/enqueue_links/enqueue_links.test.ts
index 5f233c0c1..f53f3f6ea 100644
--- a/test/core/enqueue_links/enqueue_links.test.ts
+++ b/test/core/enqueue_links/enqueue_links.test.ts
@@ -1,7 +1,7 @@
import {
type AddRequestsBatchedOptions,
cheerioCrawlerEnqueueLinks,
- MemoryStorageClient,
+ MemoryStorageBackend,
serviceLocator,
} from '@crawlee/cheerio';
import { enqueueLinks } from '@crawlee/core';
@@ -74,7 +74,7 @@ describe('enqueueLinks()', () => {
});
beforeEach(async () => {
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
});
afterAll(async () => {
diff --git a/test/core/playwright_utils.test.ts b/test/core/playwright_utils.test.ts
index de594cf6c..701ecafb8 100644
--- a/test/core/playwright_utils.test.ts
+++ b/test/core/playwright_utils.test.ts
@@ -1,7 +1,7 @@
import type { Server } from 'node:http';
import path from 'node:path';
-import { MemoryStorageClient, serviceLocator } from '@crawlee/core';
+import { MemoryStorageBackend, serviceLocator } from '@crawlee/core';
import { KeyValueStore, launchPlaywright, playwrightUtils, Request } from '@crawlee/playwright';
import type { Browser, Page } from 'playwright';
import { chromium } from 'playwright';
@@ -33,7 +33,7 @@ describe('playwrightUtils', () => {
});
beforeEach(async () => {
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
});
afterAll(async () => {
diff --git a/test/core/puppeteer_utils.test.ts b/test/core/puppeteer_utils.test.ts
index e32e1f680..0b77b49a9 100644
--- a/test/core/puppeteer_utils.test.ts
+++ b/test/core/puppeteer_utils.test.ts
@@ -1,7 +1,7 @@
import type { Server } from 'node:http';
import path from 'node:path';
-import { MemoryStorageClient, serviceLocator } from '@crawlee/core';
+import { MemoryStorageBackend, serviceLocator } from '@crawlee/core';
import { KeyValueStore, launchPuppeteer, puppeteerUtils, Request } from '@crawlee/puppeteer';
import type { Dictionary } from '@crawlee/utils';
// @ts-ignore This only throws when compiled against puppeteer 25+ (ESM only), we only import types, so its alllll gooooood
@@ -34,7 +34,7 @@ describe('puppeteerUtils', () => {
});
beforeEach(async () => {
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
});
afterAll(async () => {
diff --git a/test/core/recoverable_state.test.ts b/test/core/recoverable_state.test.ts
index 5c2519b69..b6eb67ead 100644
--- a/test/core/recoverable_state.test.ts
+++ b/test/core/recoverable_state.test.ts
@@ -1,6 +1,6 @@
import { afterEach, beforeEach, describe, expect, test, vi } from 'vitest';
-import { KeyValueStore, MemoryStorageClient, serviceLocator } from '../../packages/core/src/index.js';
+import { KeyValueStore, MemoryStorageBackend, serviceLocator } from '../../packages/core/src/index.js';
import { RecoverableState } from '../../packages/core/src/recoverable_state.js';
interface TestState {
@@ -11,7 +11,7 @@ interface TestState {
describe('RecoverableState', () => {
beforeEach(async () => {
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
});
const defaultState: TestState = {
diff --git a/test/core/request_list.test.ts b/test/core/request_list.test.ts
index c788197af..dc22882ce 100644
--- a/test/core/request_list.test.ts
+++ b/test/core/request_list.test.ts
@@ -3,7 +3,7 @@ import {
deserializeArray,
EventType,
KeyValueStore,
- MemoryStorageClient,
+ MemoryStorageBackend,
ProxyConfiguration,
Request,
RequestList,
@@ -54,7 +54,7 @@ describe('RequestList', () => {
});
beforeEach(async () => {
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
vitest.restoreAllMocks();
});
diff --git a/test/core/request_manager_tandem.test.ts b/test/core/request_manager_tandem.test.ts
index 9f29f7b27..a90a1c416 100644
--- a/test/core/request_manager_tandem.test.ts
+++ b/test/core/request_manager_tandem.test.ts
@@ -1,6 +1,6 @@
import {
log,
- MemoryStorageClient,
+ MemoryStorageBackend,
Request,
RequestList,
RequestManagerTandem,
@@ -18,7 +18,7 @@ describe('RequestManagerTandem', () => {
});
beforeEach(async () => {
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
vi.restoreAllMocks();
});
diff --git a/test/core/session_pool/session_pool.test.ts b/test/core/session_pool/session_pool.test.ts
index 398f63a23..6dedc84b6 100644
--- a/test/core/session_pool/session_pool.test.ts
+++ b/test/core/session_pool/session_pool.test.ts
@@ -2,7 +2,7 @@ import {
BaseCrawleeLogger,
EventType,
KeyValueStore,
- MemoryStorageClient,
+ MemoryStorageBackend,
serviceLocator,
Session,
SessionPool,
@@ -13,7 +13,7 @@ describe('SessionPool - testing session pool', () => {
let sessionPool: SessionPool;
beforeEach(async () => {
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
sessionPool = new SessionPool();
});
diff --git a/test/core/sitemap_request_loader.test.ts b/test/core/sitemap_request_loader.test.ts
index 7bd083ae3..b8b273e0a 100644
--- a/test/core/sitemap_request_loader.test.ts
+++ b/test/core/sitemap_request_loader.test.ts
@@ -3,7 +3,7 @@ import type { AddressInfo } from 'node:net';
import { Readable } from 'node:stream';
import { finished } from 'node:stream/promises';
-import { MemoryStorageClient, type Request, serviceLocator, SitemapRequestLoader } from '@crawlee/core';
+import { MemoryStorageBackend, type Request, serviceLocator, SitemapRequestLoader } from '@crawlee/core';
import { sleep } from '@crawlee/utils';
import express from 'express';
import { startExpressAppPromise } from '../shared/_helper.js';
@@ -262,7 +262,7 @@ afterAll(async () => {
// Fresh in-memory storage for each test
beforeEach(async () => {
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
});
describe('SitemapRequestLoader', () => {
diff --git a/test/core/storages/dataset.test.ts b/test/core/storages/dataset.test.ts
index 0c8fdc764..0c7f90d25 100644
--- a/test/core/storages/dataset.test.ts
+++ b/test/core/storages/dataset.test.ts
@@ -1,16 +1,16 @@
-import { assertJsonSerializable, Dataset, KeyValueStore, MemoryStorageClient, serviceLocator } from '@crawlee/core';
+import { assertJsonSerializable, Dataset, KeyValueStore, MemoryStorageBackend, serviceLocator } from '@crawlee/core';
import type { Dictionary } from '@crawlee/utils';
import { MAX_PAYLOAD_SIZE_BYTES } from '@apify/consts';
beforeEach(async () => {
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
});
describe('dataset', () => {
async function createDataset(id = 'some-id', name?: string) {
- const client = await serviceLocator.getStorageClient().createDatasetClient(name ? { name } : { id });
- return new Dataset({ id, name, client });
+ const client = await serviceLocator.getStorageBackend().createDatasetBackend(name ? { name } : { id });
+ return new Dataset({ id, name, backend: client });
}
beforeEach(async () => {
@@ -23,7 +23,7 @@ describe('dataset', () => {
test('should work', async () => {
const dataset = await createDataset();
- const pushDataSpy = vitest.spyOn(dataset.client, 'pushData');
+ const pushDataSpy = vitest.spyOn(dataset.backend, 'pushData');
const mockPushData = pushDataSpy.mockResolvedValueOnce(undefined);
@@ -39,7 +39,7 @@ describe('dataset', () => {
expect(mockPushData2).toHaveBeenCalledTimes(2);
expect(mockPushData2).toHaveBeenCalledWith([{ foo: 'hotel;' }, { foo: 'restaurant' }]);
- const mockDrop = vitest.spyOn(dataset.client, 'drop').mockResolvedValueOnce(undefined);
+ const mockDrop = vitest.spyOn(dataset.backend, 'drop').mockResolvedValueOnce(undefined);
await dataset.drop();
@@ -52,7 +52,7 @@ describe('dataset', () => {
const dataset = await createDataset();
- const mockPushData = vitest.spyOn(dataset.client, 'pushData');
+ const mockPushData = vitest.spyOn(dataset.backend, 'pushData');
mockPushData.mockResolvedValueOnce(undefined);
await dataset.pushData([{ foo: half }, { bar: half }]);
@@ -69,7 +69,7 @@ describe('dataset', () => {
const dataset = await createDataset();
- const mockPushData = vitest.spyOn(dataset.client, 'pushData');
+ const mockPushData = vitest.spyOn(dataset.backend, 'pushData');
mockPushData.mockResolvedValueOnce(undefined);
await dataset.pushData(data);
@@ -90,7 +90,7 @@ describe('dataset', () => {
desc: false,
};
- const mockGetData = vitest.spyOn(dataset.client, 'getData');
+ const mockGetData = vitest.spyOn(dataset.backend, 'getData');
mockGetData.mockResolvedValueOnce(expected);
const result = await dataset.getData({ limit: 2, offset: 3 });
@@ -102,7 +102,7 @@ describe('dataset', () => {
expect(result).toEqual(expected);
- vitest.spyOn(dataset.client, 'getData').mockImplementation(() => {
+ vitest.spyOn(dataset.backend, 'getData').mockImplementation(() => {
throw new Error('Cannot create a string longer than 0x3fffffe7 characters');
});
await expect(dataset.getData()).rejects.toThrow(
@@ -122,7 +122,7 @@ describe('dataset', () => {
itemCount: 14,
};
- const mockGetDataset = vitest.spyOn(dataset.client, 'getMetadata');
+ const mockGetDataset = vitest.spyOn(dataset.backend, 'getMetadata');
mockGetDataset.mockResolvedValueOnce(expected);
const result = await dataset.getInfo();
@@ -151,7 +151,7 @@ describe('dataset', () => {
desc: false,
};
- const mockGetData = vitest.spyOn(dataset.client, 'getData');
+ const mockGetData = vitest.spyOn(dataset.backend, 'getData');
mockGetData.mockResolvedValueOnce(firstResolve);
mockGetData.mockResolvedValueOnce(secondResolve);
@@ -290,7 +290,7 @@ describe('dataset', () => {
test('reduce() uses first value as memo if no memo is provided', async () => {
const dataset = await createDataset('some-id', 'some-name');
- const mockGetData = vitest.spyOn(dataset.client, 'getData');
+ const mockGetData = vitest.spyOn(dataset.backend, 'getData');
mockGetData.mockResolvedValueOnce({
items: [{ foo: 4 }, { foo: 5 }],
limit: 2,
diff --git a/test/core/storages/key_value_store.test.ts b/test/core/storages/key_value_store.test.ts
index c6dcebcbe..a10722e62 100644
--- a/test/core/storages/key_value_store.test.ts
+++ b/test/core/storages/key_value_store.test.ts
@@ -1,17 +1,17 @@
import { PassThrough } from 'node:stream';
-import { KeyValueStore, MemoryStorageClient, serviceLocator } from '@crawlee/core';
+import { KeyValueStore, MemoryStorageBackend, serviceLocator } from '@crawlee/core';
import type { Dictionary } from '@crawlee/utils';
import { toBuffer } from '@crawlee/utils';
beforeEach(async () => {
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
});
describe('KeyValueStore', () => {
async function createKeyValueStore(id = 'some-id-1', name?: string) {
- const client = await serviceLocator.getStorageClient().createKeyValueStoreClient(name ? { name } : { id });
- return new KeyValueStore({ id, name, client });
+ const client = await serviceLocator.getStorageBackend().createKeyValueStoreBackend(name ? { name } : { id });
+ return new KeyValueStore({ id, name, backend: client });
}
beforeEach(async () => {
@@ -28,7 +28,7 @@ describe('KeyValueStore', () => {
// Set record
const mockSetValue = vitest
// @ts-expect-error Accessing private property
- .spyOn(store.client, 'setValue')
+ .spyOn(store.backend, 'setValue')
.mockResolvedValueOnce(undefined);
await store.setValue('key-1', record);
@@ -43,7 +43,7 @@ describe('KeyValueStore', () => {
// Get Record
const mockGetValue = vitest
// @ts-expect-error Accessing private property
- .spyOn(store.client, 'getValue')
+ .spyOn(store.backend, 'getValue')
.mockResolvedValueOnce({
key: 'key-1',
// The client now returns raw bytes; the frontend parses them.
@@ -60,7 +60,7 @@ describe('KeyValueStore', () => {
// Record Exists
const mockRecordExists = vitest
// @ts-expect-error Accessing private property
- .spyOn(store.client, 'recordExists')
+ .spyOn(store.backend, 'recordExists')
.mockResolvedValueOnce(true);
const exists = await store.recordExists('key-1');
@@ -72,7 +72,7 @@ describe('KeyValueStore', () => {
// Delete Record
const mockDeleteValue = vitest
// @ts-expect-error Accessing private property
- .spyOn(store.client, 'deleteValue')
+ .spyOn(store.backend, 'deleteValue')
.mockResolvedValueOnce(undefined);
await store.setValue('key-1', null);
@@ -83,7 +83,7 @@ describe('KeyValueStore', () => {
// Drop store
const mockDrop = vitest
// @ts-expect-error Accessing private property
- .spyOn(store.client, 'drop')
+ .spyOn(store.backend, 'drop')
.mockResolvedValueOnce(undefined);
await store.drop();
@@ -255,7 +255,7 @@ describe('KeyValueStore', () => {
const mockSetValue = vitest
// @ts-expect-error Accessing private property
- .spyOn(store.client, 'setValue')
+ .spyOn(store.backend, 'setValue')
.mockResolvedValueOnce(undefined);
await store.setValue('key-1', 'xxxx', { contentType: 'text/plain; charset=utf-8' });
@@ -276,7 +276,7 @@ describe('KeyValueStore', () => {
const mockSetValue = vitest
// @ts-expect-error Accessing private property
- .spyOn(store.client, 'setValue')
+ .spyOn(store.backend, 'setValue')
.mockResolvedValueOnce(undefined);
await store.setValue('key-1', record);
@@ -294,7 +294,7 @@ describe('KeyValueStore', () => {
const mockSetValue = vitest
// @ts-expect-error Accessing private property
- .spyOn(store.client, 'setValue')
+ .spyOn(store.backend, 'setValue')
.mockResolvedValueOnce(undefined);
await store.setValue('key-1', 'xxxx', { contentType: 'text/plain; charset=utf-8' });
@@ -312,7 +312,7 @@ describe('KeyValueStore', () => {
const mockSetValue = vitest
// @ts-expect-error Accessing private property
- .spyOn(store.client, 'setValue')
+ .spyOn(store.backend, 'setValue')
.mockResolvedValueOnce(undefined);
const value = Buffer.from('some text value');
@@ -331,7 +331,7 @@ describe('KeyValueStore', () => {
const mockSetValue = vitest
// @ts-expect-error Accessing private property
- .spyOn(store.client, 'setValue')
+ .spyOn(store.backend, 'setValue')
.mockResolvedValueOnce(undefined);
const value = new PassThrough();
@@ -349,7 +349,7 @@ describe('KeyValueStore', () => {
});
});
- describe('round-trips through the real storage client (no content type)', () => {
+ describe('round-trips through the real storage backend (no content type)', () => {
test('object: setValue → getValue returns the same object, stored as application/json', async () => {
const store = await KeyValueStore.open();
const original = { foo: 'bar', n: 1 };
@@ -535,7 +535,7 @@ describe('KeyValueStore', () => {
const store = await createKeyValueStore('my-store-id-1');
// @ts-expect-error Accessing private property
- const mockListKeys = vitest.spyOn(store.client, 'listKeys');
+ const mockListKeys = vitest.spyOn(store.backend, 'listKeys');
mockListKeys.mockResolvedValueOnce({
items: [
{ key: 'key1', size: 1, contentType: 'application/octet-stream' },
diff --git a/test/core/storages/key_value_store_fs_roundtrip.test.ts b/test/core/storages/key_value_store_fs_roundtrip.test.ts
index 95349b5bd..a5fb1bd5d 100644
--- a/test/core/storages/key_value_store_fs_roundtrip.test.ts
+++ b/test/core/storages/key_value_store_fs_roundtrip.test.ts
@@ -1,7 +1,7 @@
import { resolve } from 'node:path';
import { KeyValueStore, serviceLocator } from '@crawlee/core';
-import { FileSystemStorageClient } from '@crawlee/fs-storage';
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
import { ensureDir, rm } from 'fs-extra';
import { cryptoRandomObjectId } from '@apify/utilities';
@@ -9,20 +9,20 @@ import { cryptoRandomObjectId } from '@apify/utilities';
/**
* Regression guard for the "centralize KVS value semantics" refactor.
*
- * The core unit tests run against {@link MemoryStorageClient}, so they never exercise the
- * default {@link FileSystemStorageClient} backend. After moving serialize/parse into the
+ * The core unit tests run against {@link MemoryStorageBackend}, so they never exercise the
+ * default {@link FileSystemStorageBackend} backend. After moving serialize/parse into the
* `KeyValueStore` frontend, the backend must be a pure byte transport — if a backend still parses
* the body itself, the frontend would double-parse (`JSON5.parse("[object Object]")`) and throw.
*
* These tests wire the real frontend to the real fs-storage backend and assert a clean round-trip.
*/
-describe('KeyValueStore frontend over FileSystemStorageClient (byte-transport contract)', () => {
+describe('KeyValueStore frontend over FileSystemStorageBackend (byte-transport contract)', () => {
const localStorageDir = resolve(import.meta.dirname, '..', 'tmp', 'fs-kvs-roundtrip', cryptoRandomObjectId(10));
beforeEach(async () => {
serviceLocator.reset();
await ensureDir(localStorageDir);
- serviceLocator.setStorageClient(new FileSystemStorageClient({ localDataDirectory: localStorageDir }));
+ serviceLocator.setStorageBackend(new FileSystemStorageBackend({ localDataDirectory: localStorageDir }));
});
afterAll(async () => {
diff --git a/test/core/storages/request_queue.test.ts b/test/core/storages/request_queue.test.ts
index 316194af6..bf4e07f4b 100644
--- a/test/core/storages/request_queue.test.ts
+++ b/test/core/storages/request_queue.test.ts
@@ -1,6 +1,6 @@
/* eslint-disable dot-notation */
-import { MemoryStorageClient, ProxyConfiguration, Request, RequestQueue, serviceLocator } from '@crawlee/core';
+import { MemoryStorageBackend, ProxyConfiguration, Request, RequestQueue, serviceLocator } from '@crawlee/core';
import { sleep } from '@crawlee/utils';
let mockHttpClient = vitest.mockObject({
@@ -25,13 +25,13 @@ beforeEach(async () => {
describe('RequestQueue remote', () => {
beforeEach(async () => {
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
vitest.clearAllMocks();
});
async function createRequestQueue(id = 'some-id', name?: string) {
- const client = await serviceLocator.getStorageClient().createRequestQueueClient(name ? { name } : { id });
- return new RequestQueue({ id, name, client }, serviceLocator.getConfiguration());
+ const client = await serviceLocator.getStorageBackend().createRequestQueueBackend(name ? { name } : { id });
+ return new RequestQueue({ id, name, backend: client }, serviceLocator.getConfiguration());
}
test('adding a request makes it fetchable; fetching again returns null while in progress', async () => {
@@ -60,7 +60,7 @@ describe('RequestQueue remote', () => {
expect(first.wasAlreadyPresent).toBe(false);
// Spy on the client only AFTER the first add so we can assert the cache prevents a second call.
- const addBatchSpy = vitest.spyOn(queue.client, 'addBatchOfRequests');
+ const addBatchSpy = vitest.spyOn(queue.backend, 'addBatchOfRequests');
const second = await queue.addRequest(requestB);
expect(second).toEqual({
@@ -280,7 +280,7 @@ describe('RequestQueue remote', () => {
test('should return correct handledCount', async () => {
const queue = await createRequestQueue('id');
- const getMock = vitest.spyOn(queue.client, 'getMetadata');
+ const getMock = vitest.spyOn(queue.backend, 'getMetadata');
getMock.mockResolvedValueOnce({
handledRequestCount: 33,
} as never);
@@ -307,7 +307,7 @@ describe('RequestQueue remote', () => {
hadMultipleClients: false,
};
- const getMock = vitest.spyOn(queue.client, 'getMetadata').mockResolvedValueOnce(expected);
+ const getMock = vitest.spyOn(queue.backend, 'getMetadata').mockResolvedValueOnce(expected);
const result = await queue.getInfo();
expect(result).toEqual(expected);
@@ -317,7 +317,7 @@ describe('RequestQueue remote', () => {
test('drop() works', async () => {
const queue = await createRequestQueue('some-id', 'some-name');
- const dropMock = vitest.spyOn(queue.client, 'drop').mockResolvedValueOnce(undefined);
+ const dropMock = vitest.spyOn(queue.backend, 'drop').mockResolvedValueOnce(undefined);
await queue.drop();
expect(dropMock).toHaveBeenCalledTimes(1);
@@ -372,7 +372,7 @@ describe('RequestQueue remote', () => {
// The in-memory client does not implement this optional hint (it has no request locking to
// tune), so attach a stub to verify the frontend's raise-only forwarding logic in isolation.
const spy = vitest.fn();
- (queue.client as any).setExpectedRequestProcessingTimeSecs = spy;
+ (queue.backend as any).setExpectedRequestProcessingTimeSecs = spy;
// First hint is forwarded.
await queue.setExpectedRequestProcessingTimeSecs(60);
@@ -422,7 +422,7 @@ describe('RequestQueue remote', () => {
describe('RequestQueue with requestsFromUrl', () => {
beforeEach(async () => {
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
vitest.restoreAllMocks();
});
diff --git a/test/core/storages/storage_aliases.test.ts b/test/core/storages/storage_aliases.test.ts
index d2932160d..5ca3a9220 100644
--- a/test/core/storages/storage_aliases.test.ts
+++ b/test/core/storages/storage_aliases.test.ts
@@ -1,13 +1,13 @@
import { resolve } from 'node:path';
-import { FileSystemStorageClient } from '@crawlee/fs-storage';
-import { Dataset, KeyValueStore, MemoryStorageClient, RequestQueue, serviceLocator } from '@crawlee/core';
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
+import { Dataset, KeyValueStore, MemoryStorageBackend, RequestQueue, serviceLocator } from '@crawlee/core';
import { ensureDir, rm } from 'fs-extra';
import { cryptoRandomObjectId } from '@apify/utilities';
beforeEach(async () => {
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
});
describe('storage aliases', () => {
@@ -146,7 +146,7 @@ describe('storage aliases', () => {
beforeEach(async () => {
serviceLocator.reset();
await ensureDir(localStorageDir);
- serviceLocator.setStorageClient(new FileSystemStorageClient({ localDataDirectory: localStorageDir }));
+ serviceLocator.setStorageBackend(new FileSystemStorageBackend({ localDataDirectory: localStorageDir }));
});
afterAll(async () => {
@@ -169,8 +169,8 @@ describe('storage aliases', () => {
// Simulate a fresh process: only the on-disk directory survives.
serviceLocator.reset();
- const client = new FileSystemStorageClient({ localDataDirectory: localStorageDir });
- serviceLocator.setStorageClient(client);
+ const client = new FileSystemStorageBackend({ localDataDirectory: localStorageDir });
+ serviceLocator.setStorageBackend(client);
// 'named-storage' is the storage's name, not its id, so it must not be resolved as an id.
await expect(client.storageExists('named-storage', 'Dataset')).resolves.toBe(false);
@@ -184,10 +184,10 @@ describe('storage aliases', () => {
// The native storage always persists the auto-assigned id to disk (in `__metadata__.json`)
// so it survives a reset. The directory is named after the storage's name, not its id.
serviceLocator.reset();
- const firstClient = new FileSystemStorageClient({
+ const firstClient = new FileSystemStorageBackend({
localDataDirectory: localStorageDir,
});
- serviceLocator.setStorageClient(firstClient);
+ serviceLocator.setStorageBackend(firstClient);
const created = await Dataset.open('some-name');
const assignedId = created.id;
@@ -199,8 +199,8 @@ describe('storage aliases', () => {
// Simulate a fresh process: only the on-disk directory survives.
serviceLocator.reset();
- const client = new FileSystemStorageClient({ localDataDirectory: localStorageDir });
- serviceLocator.setStorageClient(client);
+ const client = new FileSystemStorageBackend({ localDataDirectory: localStorageDir });
+ serviceLocator.setStorageBackend(client);
// Opening by the persisted id must find the storage, even though its directory is named
// after the name.
diff --git a/test/core/storages/storage_manager.test.ts b/test/core/storages/storage_manager.test.ts
index 2682ba61d..69b284d54 100644
--- a/test/core/storages/storage_manager.test.ts
+++ b/test/core/storages/storage_manager.test.ts
@@ -1,31 +1,20 @@
-import { Configuration, Dataset } from '@crawlee/core';
-
-import { MemoryStorageEmulator } from '../../shared/MemoryStorageEmulator';
-
-const localStorageEmulator = new MemoryStorageEmulator();
+import { Dataset, MemoryStorageBackend, serviceLocator } from '@crawlee/core';
beforeEach(async () => {
- await localStorageEmulator.init();
-});
-
-afterAll(async () => {
- await localStorageEmulator.destroy();
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
});
describe('StorageManager', () => {
test('failed openStorage call does not block subsequent calls (#3661)', async () => {
- const goodClient = Configuration.getStorageClient();
- const failingClient = {
- ...goodClient,
- datasets: () => {
- throw new Error('boom');
- },
- dataset: () => {
+ const goodBackend = serviceLocator.getStorageBackend();
+ const failingBackend = {
+ ...goodBackend,
+ createDatasetBackend: () => {
throw new Error('boom');
},
};
- await expect(Dataset.open('will-fail', { storageClient: failingClient as any })).rejects.toThrow('boom');
+ await expect(Dataset.open('will-fail', { storageBackend: failingBackend as any })).rejects.toThrow('boom');
await expect(
Promise.race([
diff --git a/test/core/storages/utils.test.ts b/test/core/storages/utils.test.ts
index f3e021486..a9bba6ece 100644
--- a/test/core/storages/utils.test.ts
+++ b/test/core/storages/utils.test.ts
@@ -1,10 +1,10 @@
import type { Dictionary } from '@crawlee/core';
-import { Configuration, KeyValueStore, MemoryStorageClient, purgeDefaultStorages, serviceLocator, useState } from '@crawlee/core';
-import type { StorageClient } from '@crawlee/types';
+import { Configuration, KeyValueStore, MemoryStorageBackend, purgeDefaultStorages, serviceLocator, useState } from '@crawlee/core';
+import type { StorageBackend } from '@crawlee/types';
describe('useState', () => {
beforeEach(async () => {
- serviceLocator.setStorageClient(new MemoryStorageClient());
+ serviceLocator.setStorageBackend(new MemoryStorageBackend());
serviceLocator.setConfiguration(new Configuration({ persistStateIntervalMillis: 1e3 }));
});
@@ -66,13 +66,13 @@ describe('purgeDefaultStorages', () => {
await new Promise((resolve) => setTimeout(resolve, 50));
purging = false;
},
- } as unknown as StorageClient;
+ } as unknown as StorageBackend;
const config = new Configuration({ purgeOnStart: true });
await Promise.all(
Array.from({ length: 3 }, async () => {
- await purgeDefaultStorages({ onlyPurgeOnce: true, client, config });
+ await purgeDefaultStorages({ onlyPurgeOnce: true, storageBackend: client, config });
if (purging) {
purgedWhileAlreadyPurging = true;
}
@@ -82,26 +82,5 @@ describe('purgeDefaultStorages', () => {
expect(purgedWhileAlreadyPurging).toBe(false);
});
- it('purges only once for configurations persisting to the same directory', async () => {
- const configA = new Configuration({ purgeOnStart: true, persistStorage: true });
- const configB = new Configuration({ purgeOnStart: true, persistStorage: true });
- const client = configA.getStorageClient();
- expect(configB.getStorageClient()).toBe(client);
-
- let purgeCount = 0;
- client.purge = async () => void purgeCount++;
-
- await purgeDefaultStorages({ onlyPurgeOnce: true, client, config: configA });
- await purgeDefaultStorages({ onlyPurgeOnce: true, client: configB.getStorageClient(), config: configB });
-
- expect(purgeCount).toBe(1);
- });
-
- it('keeps in-memory storage clients isolated per configuration', () => {
- const configA = new Configuration({ persistStorage: false });
- const configB = new Configuration({ persistStorage: false });
-
- expect(configA.getStorageClient()).not.toBe(configB.getStorageClient());
- });
});
diff --git a/test/e2e/migration/actor/main.js b/test/e2e/migration/actor/main.js
index 3667fa9f7..1176e9be2 100644
--- a/test/e2e/migration/actor/main.js
+++ b/test/e2e/migration/actor/main.js
@@ -48,7 +48,7 @@ if (workerData !== '#actor') {
},
});
- serviceLocator.getStorageClient().__purged = false;
+ serviceLocator.getStorageBackend().__purged = false;
await crawler.run(['https://crawlee.dev']);
}, mainOptions);
diff --git a/test/e2e/storage-open-return-storage-object/test.mjs b/test/e2e/storage-open-return-storage-object/test.mjs
index 2be15ea88..b5696d304 100644
--- a/test/e2e/storage-open-return-storage-object/test.mjs
+++ b/test/e2e/storage-open-return-storage-object/test.mjs
@@ -1,7 +1,7 @@
import { expect, getActorTestDir, initialize, runActor } from '../tools.mjs';
/* This test verifies that the storageObject is correctly returned when the KeyValueStore or Dataset is opened.
- * The storageObject is the result of the KeyValueStoreClient.get() or Dataset.get() methods,
+ * The storageObject is the result of the KeyValueStoreBackend.get() or Dataset.get() methods,
* containing properties such as name, id, and other custom attributes.
*/
diff --git a/website/versioned_docs/version-3.10/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.10/upgrading/upgrading_v3.md
index 39b6091c9..7f28bf10b 100644
--- a/website/versioned_docs/version-3.10/upgrading/upgrading_v3.md
+++ b/website/versioned_docs/version-3.10/upgrading/upgrading_v3.md
@@ -31,7 +31,7 @@ The [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of sever
- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative
- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package
- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods
-- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)
+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/3.10/core/interface/StorageClient)
### Installing Crawlee
diff --git a/website/versioned_docs/version-3.11/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.11/upgrading/upgrading_v3.md
index 39b6091c9..7204f43b7 100644
--- a/website/versioned_docs/version-3.11/upgrading/upgrading_v3.md
+++ b/website/versioned_docs/version-3.11/upgrading/upgrading_v3.md
@@ -31,7 +31,7 @@ The [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of sever
- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative
- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package
- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods
-- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)
+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/3.11/core/interface/StorageClient)
### Installing Crawlee
diff --git a/website/versioned_docs/version-3.12/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.12/upgrading/upgrading_v3.md
index 39b6091c9..6d9e2912f 100644
--- a/website/versioned_docs/version-3.12/upgrading/upgrading_v3.md
+++ b/website/versioned_docs/version-3.12/upgrading/upgrading_v3.md
@@ -31,7 +31,7 @@ The [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of sever
- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative
- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package
- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods
-- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)
+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/3.12/core/interface/StorageClient)
### Installing Crawlee
diff --git a/website/versioned_docs/version-3.13/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.13/upgrading/upgrading_v3.md
index 39b6091c9..fe377761b 100644
--- a/website/versioned_docs/version-3.13/upgrading/upgrading_v3.md
+++ b/website/versioned_docs/version-3.13/upgrading/upgrading_v3.md
@@ -31,7 +31,7 @@ The [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of sever
- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative
- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package
- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods
-- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)
+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/3.13/core/interface/StorageClient)
### Installing Crawlee
diff --git a/website/versioned_docs/version-3.14/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.14/upgrading/upgrading_v3.md
index 39b6091c9..0b01104db 100644
--- a/website/versioned_docs/version-3.14/upgrading/upgrading_v3.md
+++ b/website/versioned_docs/version-3.14/upgrading/upgrading_v3.md
@@ -31,7 +31,7 @@ The [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of sever
- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative
- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package
- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods
-- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)
+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/3.14/core/interface/StorageClient)
### Installing Crawlee
diff --git a/website/versioned_docs/version-3.15/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.15/upgrading/upgrading_v3.md
index 39b6091c9..53b47a54a 100644
--- a/website/versioned_docs/version-3.15/upgrading/upgrading_v3.md
+++ b/website/versioned_docs/version-3.15/upgrading/upgrading_v3.md
@@ -31,7 +31,7 @@ The [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of sever
- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative
- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package
- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods
-- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)
+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/3.15/core/interface/StorageClient)
### Installing Crawlee
diff --git a/website/versioned_docs/version-3.16/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.16/upgrading/upgrading_v3.md
index 39b6091c9..d61fc560a 100644
--- a/website/versioned_docs/version-3.16/upgrading/upgrading_v3.md
+++ b/website/versioned_docs/version-3.16/upgrading/upgrading_v3.md
@@ -31,7 +31,7 @@ The [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of sever
- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative
- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package
- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods
-- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)
+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/3.16/core/interface/StorageClient)
### Installing Crawlee
diff --git a/website/versioned_docs/version-4.0/api-typedoc.json b/website/versioned_docs/version-4.0/api-typedoc.json
index 1d4596955..11cff167a 100644
--- a/website/versioned_docs/version-4.0/api-typedoc.json
+++ b/website/versioned_docs/version-4.0/api-typedoc.json
@@ -2841,7 +2841,7 @@
},
{
"id": 18683,
- "name": "StorageClient",
+ "name": "StorageBackend",
"variant": "reference",
"kind": 4194304,
"flags": {},
@@ -12537,7 +12537,7 @@
"type": {
"type": "reference",
"target": 88,
- "name": "StorageClient",
+ "name": "StorageBackend",
"package": "@crawlee/types"
},
"inheritedFrom": {
@@ -75889,7 +75889,7 @@
},
{
"id": 7974,
- "name": "StorageClient",
+ "name": "StorageBackend",
"variant": "reference",
"kind": 4194304,
"flags": {},
@@ -99603,7 +99603,7 @@
"type": {
"type": "reference",
"target": 88,
- "name": "StorageClient",
+ "name": "StorageBackend",
"package": "@crawlee/types"
}
}
@@ -109484,7 +109484,7 @@
},
{
"id": 8744,
- "name": "StorageClient",
+ "name": "StorageBackend",
"variant": "reference",
"kind": 4194304,
"flags": {},
@@ -115689,7 +115689,7 @@
"type": {
"type": "reference",
"target": 88,
- "name": "StorageClient",
+ "name": "StorageBackend",
"package": "@crawlee/types"
},
"inheritedFrom": {
@@ -213968,7 +213968,7 @@
},
{
"id": 10781,
- "name": "StorageClient",
+ "name": "StorageBackend",
"variant": "reference",
"kind": 4194304,
"flags": {},
@@ -219754,7 +219754,7 @@
"type": {
"type": "reference",
"target": 88,
- "name": "StorageClient",
+ "name": "StorageBackend",
"package": "@crawlee/types"
},
"inheritedFrom": {
@@ -279083,7 +279083,7 @@
},
{
"kind": "code",
- "text": "```typescript\nimport { serviceLocator, BasicCrawler } from 'crawlee';\n\n// Optionally configure global services before creating crawlers\nserviceLocator.setStorageClient(myCustomClient);\n\n// Crawler uses global services\nconst crawler = new BasicCrawler({ ... });\n```"
+ "text": "```typescript\nimport { serviceLocator, BasicCrawler } from 'crawlee';\n\n// Optionally configure global services before creating crawlers\nserviceLocator.setStorageBackend(myCustomClient);\n\n// Crawler uses global services\nconst crawler = new BasicCrawler({ ... });\n```"
},
{
"kind": "text",
@@ -279201,7 +279201,7 @@
"type": {
"type": "reference",
"target": 88,
- "name": "StorageClient",
+ "name": "StorageBackend",
"package": "@crawlee/types"
}
},
@@ -279501,7 +279501,7 @@
},
{
"id": 737,
- "name": "getStorageClient",
+ "name": "getStorageBackend",
"variant": "declaration",
"kind": 2048,
"flags": {},
@@ -279517,7 +279517,7 @@
"signatures": [
{
"id": 738,
- "name": "getStorageClient",
+ "name": "getStorageBackend",
"variant": "signature",
"kind": 4096,
"flags": {},
@@ -279541,20 +279541,20 @@
"type": {
"type": "reference",
"target": 88,
- "name": "StorageClient",
+ "name": "StorageBackend",
"package": "@crawlee/types"
},
"implementationOf": {
"type": "reference",
"target": -1,
- "name": "ServiceLocatorInterface.getStorageClient"
+ "name": "ServiceLocatorInterface.getStorageBackend"
}
}
],
"implementationOf": {
"type": "reference",
"target": -1,
- "name": "ServiceLocatorInterface.getStorageClient"
+ "name": "ServiceLocatorInterface.getStorageBackend"
}
},
{
@@ -279925,7 +279925,7 @@
},
{
"id": 739,
- "name": "setStorageClient",
+ "name": "setStorageBackend",
"variant": "declaration",
"kind": 2048,
"flags": {},
@@ -279941,7 +279941,7 @@
"signatures": [
{
"id": 740,
- "name": "setStorageClient",
+ "name": "setStorageBackend",
"variant": "signature",
"kind": 4096,
"flags": {},
@@ -279991,7 +279991,7 @@
"type": {
"type": "reference",
"target": 88,
- "name": "StorageClient",
+ "name": "StorageBackend",
"package": "@crawlee/types"
}
}
@@ -280003,14 +280003,14 @@
"implementationOf": {
"type": "reference",
"target": -1,
- "name": "ServiceLocatorInterface.setStorageClient"
+ "name": "ServiceLocatorInterface.setStorageBackend"
}
}
],
"implementationOf": {
"type": "reference",
"target": -1,
- "name": "ServiceLocatorInterface.setStorageClient"
+ "name": "ServiceLocatorInterface.setStorageBackend"
}
},
{
@@ -300060,7 +300060,7 @@
"type": {
"type": "reference",
"target": 88,
- "name": "StorageClient",
+ "name": "StorageBackend",
"package": "@crawlee/types"
}
},
@@ -304413,7 +304413,7 @@
"type": {
"type": "reference",
"target": 88,
- "name": "StorageClient",
+ "name": "StorageBackend",
"package": "@crawlee/types"
}
},
@@ -308043,7 +308043,7 @@
"type": {
"type": "reference",
"target": 88,
- "name": "StorageClient",
+ "name": "StorageBackend",
"package": "@crawlee/types"
}
},
@@ -308320,7 +308320,7 @@
"type": {
"type": "reference",
"target": 88,
- "name": "StorageClient",
+ "name": "StorageBackend",
"package": "@crawlee/types"
},
"inheritedFrom": {
@@ -315553,7 +315553,7 @@
},
{
"id": 88,
- "name": "StorageClient",
+ "name": "StorageBackend",
"variant": "declaration",
"kind": 256,
"flags": {},
@@ -316317,7 +316317,7 @@
"type": {
"type": "reference",
"target": 88,
- "name": "StorageClient",
+ "name": "StorageBackend",
"package": "@crawlee/types"
}
}
@@ -322193,7 +322193,7 @@
},
{
"kind": "text",
- "text": " method on the StorageClient interface, in other words\nit will call the "
+ "text": " method on the StorageBackend interface, in other words\nit will call the "
},
{
"kind": "code",
@@ -322349,7 +322349,7 @@
},
{
"kind": "text",
- "text": " method on the StorageClient interface, in other words\nit will call the "
+ "text": " method on the StorageBackend interface, in other words\nit will call the "
},
{
"kind": "code",
@@ -322397,7 +322397,7 @@
"type": {
"type": "reference",
"target": 88,
- "name": "StorageClient",
+ "name": "StorageBackend",
"package": "@crawlee/types"
}
}
@@ -329821,7 +329821,7 @@
},
{
"id": 9493,
- "name": "StorageClient",
+ "name": "StorageBackend",
"variant": "reference",
"kind": 4194304,
"flags": {},
@@ -340929,7 +340929,7 @@
"type": {
"type": "reference",
"target": 88,
- "name": "StorageClient",
+ "name": "StorageBackend",
"package": "@crawlee/types"
},
"inheritedFrom": {
@@ -349591,7 +349591,7 @@
},
{
"id": 14625,
- "name": "StorageClient",
+ "name": "StorageBackend",
"variant": "reference",
"kind": 4194304,
"flags": {},
@@ -355452,7 +355452,7 @@
"type": {
"type": "reference",
"target": 88,
- "name": "StorageClient",
+ "name": "StorageBackend",
"package": "@crawlee/types"
},
"inheritedFrom": {
@@ -362776,7 +362776,7 @@
},
{
"id": 15406,
- "name": "StorageClient",
+ "name": "StorageBackend",
"variant": "reference",
"kind": 4194304,
"flags": {},
@@ -369779,7 +369779,7 @@
"type": {
"type": "reference",
"target": 88,
- "name": "StorageClient",
+ "name": "StorageBackend",
"package": "@crawlee/types"
},
"inheritedFrom": {
@@ -374621,14 +374621,14 @@
"implementationOf": {
"type": "reference",
"target": 92,
- "name": "storage.StorageClient.dataset"
+ "name": "storage.StorageBackend.dataset"
}
}
],
"implementationOf": {
"type": "reference",
"target": 91,
- "name": "StorageClient.dataset"
+ "name": "StorageBackend.dataset"
}
},
{
@@ -374671,14 +374671,14 @@
"implementationOf": {
"type": "reference",
"target": 90,
- "name": "storage.StorageClient.datasets"
+ "name": "storage.StorageBackend.datasets"
}
}
],
"implementationOf": {
"type": "reference",
"target": 89,
- "name": "StorageClient.datasets"
+ "name": "StorageBackend.datasets"
}
},
{
@@ -374734,14 +374734,14 @@
"implementationOf": {
"type": "reference",
"target": 97,
- "name": "storage.StorageClient.keyValueStore"
+ "name": "storage.StorageBackend.keyValueStore"
}
}
],
"implementationOf": {
"type": "reference",
"target": 96,
- "name": "StorageClient.keyValueStore"
+ "name": "StorageBackend.keyValueStore"
}
},
{
@@ -374784,14 +374784,14 @@
"implementationOf": {
"type": "reference",
"target": 95,
- "name": "storage.StorageClient.keyValueStores"
+ "name": "storage.StorageBackend.keyValueStores"
}
}
],
"implementationOf": {
"type": "reference",
"target": 94,
- "name": "StorageClient.keyValueStores"
+ "name": "StorageBackend.keyValueStores"
}
},
{
@@ -374851,14 +374851,14 @@
"implementationOf": {
"type": "reference",
"target": 106,
- "name": "storage.StorageClient.purge"
+ "name": "storage.StorageBackend.purge"
}
}
],
"implementationOf": {
"type": "reference",
"target": 105,
- "name": "StorageClient.purge"
+ "name": "StorageBackend.purge"
}
},
{
@@ -374928,14 +374928,14 @@
"implementationOf": {
"type": "reference",
"target": 102,
- "name": "storage.StorageClient.requestQueue"
+ "name": "storage.StorageBackend.requestQueue"
}
}
],
"implementationOf": {
"type": "reference",
"target": 101,
- "name": "StorageClient.requestQueue"
+ "name": "StorageBackend.requestQueue"
}
},
{
@@ -374978,14 +374978,14 @@
"implementationOf": {
"type": "reference",
"target": 100,
- "name": "storage.StorageClient.requestQueues"
+ "name": "storage.StorageBackend.requestQueues"
}
}
],
"implementationOf": {
"type": "reference",
"target": 99,
- "name": "StorageClient.requestQueues"
+ "name": "StorageBackend.requestQueues"
}
},
{
@@ -375064,14 +375064,14 @@
"implementationOf": {
"type": "reference",
"target": 110,
- "name": "storage.StorageClient.setStatusMessage"
+ "name": "storage.StorageBackend.setStatusMessage"
}
}
],
"implementationOf": {
"type": "reference",
"target": 109,
- "name": "StorageClient.setStatusMessage"
+ "name": "StorageBackend.setStatusMessage"
}
},
{
@@ -375131,14 +375131,14 @@
"implementationOf": {
"type": "reference",
"target": 108,
- "name": "storage.StorageClient.teardown"
+ "name": "storage.StorageBackend.teardown"
}
}
],
"implementationOf": {
"type": "reference",
"target": 107,
- "name": "StorageClient.teardown"
+ "name": "StorageBackend.teardown"
}
}
],
@@ -375192,7 +375192,7 @@
{
"type": "reference",
"target": 88,
- "name": "StorageClient",
+ "name": "StorageBackend",
"package": "@crawlee/types"
}
]
@@ -378206,7 +378206,7 @@
},
{
"id": 12900,
- "name": "StorageClient",
+ "name": "StorageBackend",
"variant": "reference",
"kind": 4194304,
"flags": {},
@@ -401345,7 +401345,7 @@
"type": {
"type": "reference",
"target": 88,
- "name": "StorageClient",
+ "name": "StorageBackend",
"package": "@crawlee/types"
},
"inheritedFrom": {
@@ -408277,7 +408277,7 @@
"type": {
"type": "reference",
"target": 88,
- "name": "StorageClient",
+ "name": "StorageBackend",
"package": "@crawlee/types"
},
"inheritedFrom": {
@@ -420298,7 +420298,7 @@
},
{
"id": 11778,
- "name": "StorageClient",
+ "name": "StorageBackend",
"variant": "reference",
"kind": 4194304,
"flags": {},
@@ -430993,7 +430993,7 @@
"type": {
"type": "reference",
"target": 88,
- "name": "StorageClient",
+ "name": "StorageBackend",
"package": "@crawlee/types"
},
"inheritedFrom": {
@@ -435951,7 +435951,7 @@
},
{
"id": 19564,
- "name": "StorageClient",
+ "name": "StorageBackend",
"variant": "reference",
"kind": 4194304,
"flags": {},
@@ -455058,23 +455058,23 @@
},
"88": {
"sourceFileName": "../packages/types/src/storages.ts",
- "qualifiedName": "StorageClient"
+ "qualifiedName": "StorageBackend"
},
"89": {
"sourceFileName": "../packages/types/src/storages.ts",
- "qualifiedName": "StorageClient.datasets"
+ "qualifiedName": "StorageBackend.datasets"
},
"90": {
"sourceFileName": "../packages/types/src/storages.ts",
- "qualifiedName": "StorageClient.datasets"
+ "qualifiedName": "StorageBackend.datasets"
},
"91": {
"sourceFileName": "../packages/types/src/storages.ts",
- "qualifiedName": "StorageClient.dataset"
+ "qualifiedName": "StorageBackend.dataset"
},
"92": {
"sourceFileName": "../packages/types/src/storages.ts",
- "qualifiedName": "StorageClient.dataset"
+ "qualifiedName": "StorageBackend.dataset"
},
"93": {
"sourceFileName": "../packages/types/src/storages.ts",
@@ -455082,19 +455082,19 @@
},
"94": {
"sourceFileName": "../packages/types/src/storages.ts",
- "qualifiedName": "StorageClient.keyValueStores"
+ "qualifiedName": "StorageBackend.keyValueStores"
},
"95": {
"sourceFileName": "../packages/types/src/storages.ts",
- "qualifiedName": "StorageClient.keyValueStores"
+ "qualifiedName": "StorageBackend.keyValueStores"
},
"96": {
"sourceFileName": "../packages/types/src/storages.ts",
- "qualifiedName": "StorageClient.keyValueStore"
+ "qualifiedName": "StorageBackend.keyValueStore"
},
"97": {
"sourceFileName": "../packages/types/src/storages.ts",
- "qualifiedName": "StorageClient.keyValueStore"
+ "qualifiedName": "StorageBackend.keyValueStore"
},
"98": {
"sourceFileName": "../packages/types/src/storages.ts",
@@ -455102,19 +455102,19 @@
},
"99": {
"sourceFileName": "../packages/types/src/storages.ts",
- "qualifiedName": "StorageClient.requestQueues"
+ "qualifiedName": "StorageBackend.requestQueues"
},
"100": {
"sourceFileName": "../packages/types/src/storages.ts",
- "qualifiedName": "StorageClient.requestQueues"
+ "qualifiedName": "StorageBackend.requestQueues"
},
"101": {
"sourceFileName": "../packages/types/src/storages.ts",
- "qualifiedName": "StorageClient.requestQueue"
+ "qualifiedName": "StorageBackend.requestQueue"
},
"102": {
"sourceFileName": "../packages/types/src/storages.ts",
- "qualifiedName": "StorageClient.requestQueue"
+ "qualifiedName": "StorageBackend.requestQueue"
},
"103": {
"sourceFileName": "../packages/types/src/storages.ts",
@@ -455126,27 +455126,27 @@
},
"105": {
"sourceFileName": "../packages/types/src/storages.ts",
- "qualifiedName": "StorageClient.purge"
+ "qualifiedName": "StorageBackend.purge"
},
"106": {
"sourceFileName": "../packages/types/src/storages.ts",
- "qualifiedName": "StorageClient.purge"
+ "qualifiedName": "StorageBackend.purge"
},
"107": {
"sourceFileName": "../packages/types/src/storages.ts",
- "qualifiedName": "StorageClient.teardown"
+ "qualifiedName": "StorageBackend.teardown"
},
"108": {
"sourceFileName": "../packages/types/src/storages.ts",
- "qualifiedName": "StorageClient.teardown"
+ "qualifiedName": "StorageBackend.teardown"
},
"109": {
"sourceFileName": "../packages/types/src/storages.ts",
- "qualifiedName": "StorageClient.setStatusMessage"
+ "qualifiedName": "StorageBackend.setStatusMessage"
},
"110": {
"sourceFileName": "../packages/types/src/storages.ts",
- "qualifiedName": "StorageClient.setStatusMessage"
+ "qualifiedName": "StorageBackend.setStatusMessage"
},
"111": {
"sourceFileName": "../packages/types/src/storages.ts",
@@ -455158,7 +455158,7 @@
},
"113": {
"sourceFileName": "../packages/types/src/storages.ts",
- "qualifiedName": "StorageClient.stats"
+ "qualifiedName": "StorageBackend.stats"
},
"114": {
"sourceFileName": "../packages/types/src/storages.ts",
@@ -456898,19 +456898,19 @@
},
"737": {
"sourceFileName": "../packages/core/src/service_locator.ts",
- "qualifiedName": "ServiceLocator.getStorageClient"
+ "qualifiedName": "ServiceLocator.getStorageBackend"
},
"738": {
"sourceFileName": "../packages/core/src/service_locator.ts",
- "qualifiedName": "ServiceLocator.getStorageClient"
+ "qualifiedName": "ServiceLocator.getStorageBackend"
},
"739": {
"sourceFileName": "../packages/core/src/service_locator.ts",
- "qualifiedName": "ServiceLocator.setStorageClient"
+ "qualifiedName": "ServiceLocator.setStorageBackend"
},
"740": {
"sourceFileName": "../packages/core/src/service_locator.ts",
- "qualifiedName": "ServiceLocator.setStorageClient"
+ "qualifiedName": "ServiceLocator.setStorageBackend"
},
"741": {
"sourceFileName": "../packages/core/src/service_locator.ts",
@@ -478774,7 +478774,7 @@
},
"7974": {
"sourceFileName": "../packages/core/src/index.ts",
- "qualifiedName": "StorageClient"
+ "qualifiedName": "StorageBackend"
},
"7975": {
"sourceFileName": "../packages/core/src/index.ts",
@@ -480314,7 +480314,7 @@
},
"8744": {
"sourceFileName": "../packages/core/src/index.ts",
- "qualifiedName": "StorageClient"
+ "qualifiedName": "StorageBackend"
},
"8745": {
"sourceFileName": "../packages/core/src/index.ts",
@@ -481770,7 +481770,7 @@
},
"9493": {
"sourceFileName": "../packages/core/src/index.ts",
- "qualifiedName": "StorageClient"
+ "qualifiedName": "StorageBackend"
},
"9494": {
"sourceFileName": "../packages/core/src/index.ts",
@@ -484266,7 +484266,7 @@
},
"10781": {
"sourceFileName": "../packages/core/src/index.ts",
- "qualifiedName": "StorageClient"
+ "qualifiedName": "StorageBackend"
},
"10782": {
"sourceFileName": "../packages/core/src/index.ts",
@@ -486306,7 +486306,7 @@
},
"11778": {
"sourceFileName": "../packages/core/src/index.ts",
- "qualifiedName": "StorageClient"
+ "qualifiedName": "StorageBackend"
},
"11779": {
"sourceFileName": "../packages/core/src/index.ts",
@@ -488542,7 +488542,7 @@
},
"12900": {
"sourceFileName": "../packages/core/src/index.ts",
- "qualifiedName": "StorageClient"
+ "qualifiedName": "StorageBackend"
},
"12901": {
"sourceFileName": "../packages/core/src/index.ts",
@@ -492446,7 +492446,7 @@
},
"14625": {
"sourceFileName": "../packages/core/src/index.ts",
- "qualifiedName": "StorageClient"
+ "qualifiedName": "StorageBackend"
},
"14626": {
"sourceFileName": "../packages/core/src/index.ts",
@@ -494054,7 +494054,7 @@
},
"15406": {
"sourceFileName": "../packages/core/src/index.ts",
- "qualifiedName": "StorageClient"
+ "qualifiedName": "StorageBackend"
},
"15407": {
"sourceFileName": "../packages/core/src/index.ts",
@@ -504494,7 +504494,7 @@
},
"18683": {
"sourceFileName": "../packages/core/src/index.ts",
- "qualifiedName": "StorageClient"
+ "qualifiedName": "StorageBackend"
},
"18684": {
"sourceFileName": "../packages/core/src/index.ts",
@@ -507370,7 +507370,7 @@
},
"19564": {
"sourceFileName": "../packages/types/src/storages.ts",
- "qualifiedName": "StorageClient"
+ "qualifiedName": "StorageBackend"
},
"19568": {
"sourceFileName": "../packages/types/src/utility-types.ts",
@@ -508031,4 +508031,4 @@
"24": 18953
}
}
-}
\ No newline at end of file
+}
diff --git a/website/versioned_docs/version-4.0/guides/request_storage.mdx b/website/versioned_docs/version-4.0/guides/request_storage.mdx
index 8da5489b5..7d28f6ba4 100644
--- a/website/versioned_docs/version-4.0/guides/request_storage.mdx
+++ b/website/versioned_docs/version-4.0/guides/request_storage.mdx
@@ -143,4 +143,4 @@ import { purgeDefaultStorages } from 'crawlee';
await purgeDefaultStorages();
```
-Calling this function will clean up the default request storage directory (and also the request list stored in default key-value store). This is a shortcut for running (optional) `purge` method on the <ApiLink to="core/interface/StorageClient">`StorageClient`</ApiLink> interface, in other words it will call the `purge` method of the underlying storage implementation we are currently using. You can make sure the storage is purged only once for a given execution context if you set `onlyPurgeOnce` to `true` in the `options` object.
+Calling this function will clean up the default request storage directory (and also the request list stored in default key-value store). This is a shortcut for running (optional) `purge` method on the <ApiLink to="core/interface/StorageBackend">`StorageBackend`</ApiLink> interface, in other words it will call the `purge` method of the underlying storage implementation we are currently using. You can make sure the storage is purged only once for a given execution context if you set `onlyPurgeOnce` to `true` in the `options` object.
diff --git a/website/versioned_docs/version-4.0/guides/result_storage.mdx b/website/versioned_docs/version-4.0/guides/result_storage.mdx
index eff313aef..92ea25bfd 100644
--- a/website/versioned_docs/version-4.0/guides/result_storage.mdx
+++ b/website/versioned_docs/version-4.0/guides/result_storage.mdx
@@ -110,4 +110,4 @@ import { purgeDefaultStorages } from 'crawlee';
await purgeDefaultStorages();
```
-Calling this function will clean up the default results storage directories except the `INPUT` key in default key-value store directory. This is a shortcut for running (optional) `purge` method on the <ApiLink to="core/interface/StorageClient">`StorageClient`</ApiLink> interface, in other words it will call the `purge` method of the underlying storage implementation we are currently using. In addition, this method will make sure the storage is purged only once for a given execution context, so it is safe to call it multiple times.
+Calling this function will clean up the default results storage directories except the `INPUT` key in default key-value store directory. This is a shortcut for running (optional) `purge` method on the <ApiLink to="core/interface/StorageBackend">`StorageBackend`</ApiLink> interface, in other words it will call the `purge` method of the underlying storage implementation we are currently using. In addition, this method will make sure the storage is purged only once for a given execution context, so it is safe to call it multiple times.
diff --git a/website/versioned_docs/version-4.0/upgrading/upgrading_v3.md b/website/versioned_docs/version-4.0/upgrading/upgrading_v3.md
index 39b6091c9..d61fc560a 100644
--- a/website/versioned_docs/version-4.0/upgrading/upgrading_v3.md
+++ b/website/versioned_docs/version-4.0/upgrading/upgrading_v3.md
@@ -31,7 +31,7 @@ The [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of sever
- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative
- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package
- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods
-- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)
+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/3.16/core/interface/StorageClient)
### Installing Crawlee