Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

feat: Implement storage aliases (#3636)

Jan Buchar7edc79397ccf
diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.md
index 5080405e4..ac2b19c05 100644
--- a/docs/upgrading/upgrading_v4.md
+++ b/docs/upgrading/upgrading_v4.md
@@ -532,6 +532,52 @@ If you implemented a custom `StorageClient`, you need to:
2. Replace the six getter methods (`dataset`, `datasets`, `keyValueStore`, `keyValueStores`, `requestQueue`, `requestQueues`) with three async factory methods (`createDatasetClient`, `createKeyValueStoreClient`, `createRequestQueueClient`). Each factory should handle both opening an existing storage and creating a new one.
3. Apply the sub-client renames listed above (`get` → `getMetadata`, `delete` → `drop`, etc.) and implement the new `purge()` method.
+## Multiple crawler instances use separate default request queues
+
+In v3, every `BasicCrawler` (or subclass) that didn't receive an explicit `requestQueue` option would open the same default request queue. If you created two crawlers in the same process, they would silently share a queue — leading to request collisions and hard-to-debug deduplication issues.
+
+In v4, only the **first** crawler instance uses the default request queue. Each subsequent instance automatically gets its own queue via an internal alias (e.g. `__default_1__`, `__default_2__`, etc.). This means multiple crawlers can safely coexist without interfering with each other's requests.
+
+If you explicitly pass a `requestQueue` (or `requestManager`) to the crawler, that queue is used as-is regardless of instance order.
+
+## Repeated `run()` calls use `purge()` instead of `drop()` + recreate
+
+When calling `crawler.run()` multiple times on the same crawler instance, v3 would drop the default request queue and create a fresh one between runs. In v4, the crawler **purges** the queue instead — clearing all requests and resetting internal counters, but keeping the same queue object. This is more efficient and avoids edge cases around stale references.
+
+The new `purge()` method is available on `RequestProvider` (the base class for `RequestQueue`) and is also defined as an optional method on the `IRequestManager` interface.
+
+By default, only queues that the crawler created itself (the "owned" queue) are purged between runs — a user-supplied queue is never touched unless you explicitly opt in. The `purgeRequestQueue` option in `CrawlerRunOptions` controls this behavior:
+
+| `purgeRequestQueue` value | Owned queue (auto-created) | User-supplied queue |
+|---|---|---|
+| omitted (default) | Purged | Not purged |
+| `true` | Purged | Purged |
+| `false` | Not purged | Not purged |
+
+```typescript
+// The purge happens automatically between run() calls:
+const crawler = new BasicCrawler({ requestHandler: async ({ request }) => { /* ... */ } });
+await crawler.run(['https://example.com/a', 'https://example.com/b']);
+// Queue is purged here, so the same URLs can be processed again:
+await crawler.run(['https://example.com/a', 'https://example.com/c']);
+```
+
+You can opt out of the automatic purge by passing `purgeRequestQueue: false`:
+
+```typescript
+await crawler.run(urls, { purgeRequestQueue: false });
+```
+
+If you supplied your own `requestQueue` and want it purged between runs, pass `purgeRequestQueue: true` explicitly:
+
+```typescript
+const queue = await RequestQueue.open('my-queue');
+const crawler = new BasicCrawler({ requestQueue: queue, requestHandler: async () => { /* ... */ } });
+await crawler.run(['https://example.com/first']);
+// Explicitly purge the user-supplied queue before the second run:
+await crawler.run(['https://example.com/second'], { purgeRequestQueue: true });
+```
+
## Storage `.open()` now also accepts `{ id?, name? }`
`Dataset.open()`, `KeyValueStore.open()`, and `RequestQueue.open()` previously accepted a single `idOrName?: string` parameter. This was ambiguous — callers couldn't express whether they were opening a storage by its ID or by name.
diff --git a/packages/basic-crawler/src/internals/basic-crawler.ts b/packages/basic-crawler/src/internals/basic-crawler.ts
index 16ac49615..40ed11e49 100644
--- a/packages/basic-crawler/src/internals/basic-crawler.ts
+++ b/packages/basic-crawler/src/internals/basic-crawler.ts
@@ -542,6 +542,13 @@ export class BasicCrawler<
*/
private static useStateCrawlerIds = new Set<string>();
+ /**
+ * Tracks the number of crawler instances created. The first crawler uses the default
+ * request queue; subsequent ones get their own queue via a unique alias so they don't
+ * collide.
+ */
+ private static instanceCount = 0;
+
/**
* A reference to the underlying {@apilink Statistics} class that collects and logs run statistics for requests.
*/
@@ -579,6 +586,13 @@ export class BasicCrawler<
*/
private ownedSessionPool?: SessionPool;
+ /**
+ * Set when the crawler constructed its own {@apilink RequestQueue} (no `requestQueue`/`requestManager`
+ * option was provided). The owned queue is purged (not dropped) between repeated `run()` calls.
+ * A user-supplied queue is never purged by the crawler.
+ */
+ private ownedRequestQueue?: RequestProvider;
+
/**
* A reference to the underlying {@apilink AutoscaledPool} class that manages the concurrency of the crawler.
* > *NOTE:* This property is only initialized after calling the {@apilink BasicCrawler.run|`crawler.run()`} function.
@@ -678,6 +692,7 @@ export class BasicCrawler<
private _experimentWarnings: Partial<Record<keyof CrawlerExperiments, boolean>> = {};
private readonly crawlerId: string;
private readonly hasExplicitId: boolean;
+ private readonly crawlerInstanceIndex: number;
private readonly contextPipelineOptions: {
contextPipelineBuilder?: () => ContextPipeline<CrawlingContext, Context>;
extendContext?: (context: Context) => Awaitable<ContextExtension>;
@@ -820,6 +835,7 @@ export class BasicCrawler<
this.hasExplicitId = id !== undefined;
// Store the user-provided ID, or generate a unique one for tracking purposes (not for state key)
this.crawlerId = id ?? cryptoRandomObjectId();
+ this.crawlerInstanceIndex = BasicCrawler.instanceCount++;
if (requestManager !== undefined) {
if (requestList !== undefined || requestQueue !== undefined) {
@@ -1301,19 +1317,22 @@ export class BasicCrawler<
);
}
- const { purgeRequestQueue = true, ...addRequestsOptions } = options ?? {};
+ const { purgeRequestQueue, ...addRequestsOptions } = options ?? {};
if (this.hasFinishedBefore) {
// When executing the run method for the second time explicitly,
- // we need to purge the default RQ to allow processing the same requests again - this is important so users can
+ // we need to purge the RQ to allow processing the same requests again — this is important so users can
// pass in failed requests back to the `crawler.run()`, otherwise they would be considered as handled and
- // ignored - as a failed requests is still handled.
- const isDefaultQueue = this.requestQueue?.name === 'default';
- if (isDefaultQueue && purgeRequestQueue && this.requestQueue) {
- await this.requestQueue.drop();
- this.requestQueue = await this._getRequestQueue();
- this.requestManager = undefined;
- await this.initializeRequestManager();
+ // ignored — as a failed request is still handled.
+ // By default (purgeRequestQueue unset or true), only the queue we created ourselves (ownedRequestQueue) is purged.
+ // When `purgeRequestQueue` is explicitly `true`, we also purge a user-supplied queue.
+ // When `purgeRequestQueue` is explicitly `false`, nothing is purged.
+ const shouldPurge = purgeRequestQueue !== false;
+ const queueToPurge = this.ownedRequestQueue ?? (purgeRequestQueue === true ? this.requestQueue : undefined);
+
+ if (queueToPurge && shouldPurge) {
+ await queueToPurge.purge();
+ this.handledRequestsCount = 0; // This would've been reset by this._init() further down below, but at that point `handledRequestsCount` could prevent `addRequests` from adding the initial requests
}
this.stats.reset();
@@ -1445,6 +1464,7 @@ export class BasicCrawler<
if (!this.requestQueue) {
this.requestQueue = await this._getRequestQueue();
+ this.ownedRequestQueue = this.requestQueue;
this.requestManager = undefined;
}
@@ -2348,6 +2368,11 @@ export class BasicCrawler<
}
private async _getRequestQueue() {
+ // The first crawler instance uses the default queue (null identifier);
+ // subsequent instances get their own queue via a unique alias so they don't collide.
+ const identifier =
+ this.crawlerInstanceIndex === 0 ? null : { alias: `__default_${this.crawlerInstanceIndex}__` };
+
// Check if it's explicitly disabled
// oxlint-disable-next-line typescript/no-deprecated -- still honored for opt-out until the flag is removed
if (this.experiments.requestLocking === false) {
@@ -2358,10 +2383,10 @@ export class BasicCrawler<
this._experimentWarnings.requestLocking = true;
}
- return RequestQueueV1.open(null, { config: serviceLocator.getConfiguration() });
+ return RequestQueueV1.open(identifier, { config: serviceLocator.getConfiguration() });
}
- return RequestQueue.open(null, { config: serviceLocator.getConfiguration() });
+ return RequestQueue.open(identifier, { config: serviceLocator.getConfiguration() });
}
private requestMatchesEnqueueStrategy(request: Request) {
@@ -2435,9 +2460,14 @@ export interface CrawlerAddRequestsResult extends AddRequestsBatchedResult {}
export interface CrawlerRunOptions extends CrawlerAddRequestsOptions {
/**
- * Whether to purge the RequestQueue before running the crawler again. Defaults to true, so it is possible to reprocess failed requests.
- * When disabled, only new requests will be considered. Note that even a failed request is considered as handled.
- * @default true
+ * Controls whether the request queue is purged between repeated `run()` calls on the same crawler instance.
+ * Purging clears all requests and resets internal counters, allowing the same URLs to be processed again.
+ *
+ * - **`undefined`** (default) — only the crawler's own (auto-created) queue is purged.
+ * A user-supplied `requestQueue` is left untouched.
+ * - **`true`** — the queue is always purged, even if it was supplied by the user.
+ * - **`false`** — nothing is purged. Only genuinely new requests will be processed;
+ * note that even a failed request is considered handled.
*/
purgeRequestQueue?: boolean;
}
diff --git a/packages/core/src/storages/request_provider.ts b/packages/core/src/storages/request_provider.ts
index 6b2ee8f9e..abed5d494 100644
--- a/packages/core/src/storages/request_provider.ts
+++ b/packages/core/src/storages/request_provider.ts
@@ -106,6 +106,14 @@ export interface IRequestManager {
addRequest(requestLike: Source, options?: RequestQueueOperationOptions): Promise<RequestQueueOperationInfo>;
addRequestsBatched(requests: RequestsLike, options?: AddRequestsBatchedOptions): Promise<AddRequestsBatchedResult>;
+
+ /**
+ * Remove all requests from the queue but keep the queue itself, resetting it
+ * so it can be reused (e.g. across multiple `crawler.run()` calls).
+ *
+ * Implementations that do not support purging may leave this `undefined`.
+ */
+ purge?(): Promise<void>;
}
export abstract class RequestProvider implements IStorage, IRequestManager {
@@ -808,6 +816,28 @@ export abstract class RequestProvider implements IStorage, IRequestManager {
serviceLocator.getStorageInstanceManager().removeFromCache(this);
}
+ /**
+ * Remove all requests from the queue but keep the queue itself, resetting it
+ * so it can be reused (e.g. across multiple `crawler.run()` calls).
+ */
+ async purge(): Promise<void> {
+ checkStorageAccess();
+
+ await this.client.purge();
+
+ // Reset in-memory bookkeeping so the queue behaves as if freshly opened.
+ this.assumedTotalCount = 0;
+ this.assumedHandledCount = 0;
+ this.initialCount = 0;
+ this.initialHandledCount = 0;
+ this.queueHeadIds.clear();
+ this.requestCache.clear();
+ this.recentlyHandledRequestsCache.clear();
+ this.lastActivity = new Date();
+ this.isFinishedCalledWhileHeadWasNotEmpty = 0;
+ this.inProgressRequestBatchCount = 0;
+ }
+
/**
* @inheritdoc
*/
diff --git a/packages/core/src/storages/storage_instance_manager.ts b/packages/core/src/storages/storage_instance_manager.ts
index ee8f73c3b..7bf85b13c 100644
--- a/packages/core/src/storages/storage_instance_manager.ts
+++ b/packages/core/src/storages/storage_instance_manager.ts
@@ -119,6 +119,33 @@ class StorageCache {
}
}
+ /**
+ * Ensure that the same string is not used as both a name and an alias for the same
+ * storage class + backend combination. Mirrors crawlee-python's `_check_name_alias_conflict`.
+ */
+ checkNameAliasConflict<T extends IStorage>(
+ cls: Constructor<T>,
+ { name, alias, clientCacheKey }: { name?: string; alias?: string; clientCacheKey: Hashable },
+ ): void {
+ if (alias) {
+ const existingByName = this.byName.get(cls)?.get(alias)?.get(clientCacheKey);
+ if (existingByName) {
+ throw new Error(
+ `Cannot open storage with alias "${alias}" because a named storage with the same identifier already exists.`,
+ );
+ }
+ }
+ if (name) {
+ const existingByAlias = this.byAlias.get(cls)?.get(name)?.get(clientCacheKey);
+ if (existingByAlias) {
+ throw new Error(
+ `Cannot open storage with name "${name}" because an alias storage with the same identifier already exists.` +
+ ` If you meant to open the alias storage, use { alias: "${name}" } instead.`,
+ );
+ }
+ }
+ }
+
/** Iterate all cached instances across all storage types. */
*allValues(): IterableIterator<IStorage> {
const seen = new Set<IStorage>();
@@ -222,6 +249,9 @@ export class StorageInstanceManager {
if (cached) return cached;
}
+ // Prevent the same string from being used as both a name and an alias.
+ this.cache.checkNameAliasConflict(cls, { name, alias, clientCacheKey });
+
// Cache miss — create the sub-client and storage instance.
const subClient = await clientOpener();
const storageInfo = await (
@@ -299,6 +329,7 @@ export interface DefaultStorageIdentifier {
* - `string` → resolved via `storageExists` (ID-first, then name)
* - `{ id }` → `{ id }`
* - `{ name }` → `{ name }`
+ * - `{ alias }` → `{ alias }`
*/
export async function resolveStorageIdentifier(
identifier: string | StorageIdentifier | null | undefined,
@@ -324,6 +355,10 @@ export async function resolveStorageIdentifier(
return { name: identifier.name };
}
+ if ('alias' in identifier && identifier.alias) {
+ return { alias: identifier.alias };
+ }
+
// Empty object — treated as default storage.
return { alias: DEFAULT_STORAGE_ALIAS };
}
diff --git a/packages/memory-storage/src/cache-helpers.ts b/packages/memory-storage/src/cache-helpers.ts
index 3d213e1ea..c0076b6d9 100644
--- a/packages/memory-storage/src/cache-helpers.ts
+++ b/packages/memory-storage/src/cache-helpers.ts
@@ -13,9 +13,12 @@ import { type MemoryStorage } from './memory-storage.js';
const uuidRegex = /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/i;
export async function findOrCacheDatasetByPossibleId(client: MemoryStorage, entryNameOrId: string) {
- // First check memory cache
+ // First check memory cache — match by id, name, or directoryName (which covers alias lookups)
const found = client.datasetClientCache.find(
- (store) => store.id === entryNameOrId || store.name?.toLowerCase() === entryNameOrId.toLowerCase(),
+ (store) =>
+ store.id === entryNameOrId ||
+ store.name?.toLowerCase() === entryNameOrId.toLowerCase() ||
+ store.directoryName.toLowerCase() === entryNameOrId.toLowerCase(),
);
if (found) {
@@ -116,9 +119,12 @@ export async function findOrCacheDatasetByPossibleId(client: MemoryStorage, entr
}
export async function findOrCacheKeyValueStoreByPossibleId(client: MemoryStorage, entryNameOrId: string) {
- // First check memory cache
+ // First check memory cache — match by id, name, or directoryName (which covers alias lookups)
const found = client.keyValueStoreCache.find(
- (store) => store.id === entryNameOrId || store.name?.toLowerCase() === entryNameOrId.toLowerCase(),
+ (store) =>
+ store.id === entryNameOrId ||
+ store.name?.toLowerCase() === entryNameOrId.toLowerCase() ||
+ store.directoryName.toLowerCase() === entryNameOrId.toLowerCase(),
);
if (found) {
@@ -270,9 +276,12 @@ export async function findOrCacheKeyValueStoreByPossibleId(client: MemoryStorage
}
export async function findRequestQueueByPossibleId(client: MemoryStorage, entryNameOrId: string) {
- // First check memory cache
+ // First check memory cache — match by id, name, or directoryName (which covers alias lookups)
const found = client.requestQueueCache.find(
- (store) => store.id === entryNameOrId || store.name?.toLowerCase() === entryNameOrId.toLowerCase(),
+ (store) =>
+ store.id === entryNameOrId ||
+ store.name?.toLowerCase() === entryNameOrId.toLowerCase() ||
+ store.directoryName.toLowerCase() === entryNameOrId.toLowerCase(),
);
if (found) {
diff --git a/packages/memory-storage/src/memory-storage.ts b/packages/memory-storage/src/memory-storage.ts
index 0ccc6da2f..c0c20a8ec 100644
--- a/packages/memory-storage/src/memory-storage.ts
+++ b/packages/memory-storage/src/memory-storage.ts
@@ -103,23 +103,33 @@ export class MemoryStorage implements storage.StorageClient {
return `MemoryStorage:${resolve(this.localDataDirectory)}`;
}
- private static resolveStorageName(options: { id?: string; name?: string; alias?: string }): string | undefined {
- const name = 'alias' in options ? options.alias : (options.name ?? options.id);
+ private static resolveStorageKey(options: { id?: string; name?: string; alias?: string }): {
+ isAlias: boolean;
+ directoryKey: string | undefined;
+ } {
+ const isAlias = 'alias' in options && !!options.alias;
+ const rawKey = isAlias ? options.alias : (options.name ?? options.id);
// Normalize the internal __default__ alias to the user-facing 'default' name.
- return name === '__default__' ? 'default' : name;
+ const directoryKey = rawKey === '__default__' ? 'default' : rawKey;
+ return { isAlias, directoryKey };
}
async createDatasetClient(options: storage.CreateDatasetClientOptions = {}): Promise<storage.DatasetClient> {
- const name = MemoryStorage.resolveStorageName(options);
+ const { isAlias, directoryKey } = MemoryStorage.resolveStorageKey(options);
- if (name) {
- const found = await findOrCacheDatasetByPossibleId(this, name);
+ if (directoryKey) {
+ const found = await findOrCacheDatasetByPossibleId(this, directoryKey);
if (found) {
return found;
}
}
- const newStore = new DatasetClient({ name, baseStorageDirectory: this.datasetsDirectory, client: this });
+ const newStore = new DatasetClient({
+ name: isAlias ? undefined : directoryKey,
+ directoryName: directoryKey,
+ baseStorageDirectory: this.datasetsDirectory,
+ client: this,
+ });
this.datasetClientCache.push(newStore);
// Schedule the worker to write to the disk
@@ -144,17 +154,18 @@ export class MemoryStorage implements storage.StorageClient {
async createKeyValueStoreClient(
options: storage.CreateKeyValueStoreClientOptions = {},
): Promise<storage.KeyValueStoreClient> {
- const name = MemoryStorage.resolveStorageName(options);
+ const { isAlias, directoryKey } = MemoryStorage.resolveStorageKey(options);
- if (name) {
- const found = await findOrCacheKeyValueStoreByPossibleId(this, name);
+ if (directoryKey) {
+ const found = await findOrCacheKeyValueStoreByPossibleId(this, directoryKey);
if (found) {
return found;
}
}
const newStore = new KeyValueStoreClient({
- name,
+ name: isAlias ? undefined : directoryKey,
+ directoryName: directoryKey,
baseStorageDirectory: this.keyValueStoresDirectory,
client: this,
});
@@ -182,17 +193,18 @@ export class MemoryStorage implements storage.StorageClient {
async createRequestQueueClient(
options: storage.CreateRequestQueueClientOptions = {},
): Promise<storage.RequestQueueClient> {
- const name = MemoryStorage.resolveStorageName(options);
+ const { isAlias, directoryKey } = MemoryStorage.resolveStorageKey(options);
- if (name) {
- const found = await findRequestQueueByPossibleId(this, name);
+ if (directoryKey) {
+ const found = await findRequestQueueByPossibleId(this, directoryKey);
if (found) {
return found;
}
}
const newStore = new RequestQueueClient({
- name,
+ name: isAlias ? undefined : directoryKey,
+ directoryName: directoryKey,
baseStorageDirectory: this.requestQueuesDirectory,
client: this,
});
@@ -238,14 +250,26 @@ export class MemoryStorage implements storage.StorageClient {
return false;
}
- // Check in-memory cache first
+ // Check in-memory cache by actual storage ID
if (clients.some((store) => store.id === id)) {
return true;
}
- // Check if a directory with that ID exists on disk
+ // Check if a directory with that ID exists on disk.
+ // Only consider directories whose name matches the queried ID — this avoids
+ // false positives for alias-created directories (e.g. a directory named 'asdf'
+ // created via `{ alias: 'asdf' }` should not make `storageExists('asdf')` return true,
+ // since the actual storage ID is a UUID, not the alias string).
try {
await access(resolve(baseDir, id));
+
+ // If the directory exists but a cached client already owns this directory
+ // under a different ID, this is not a match.
+ const cachedClients = clients as { id: string; directoryName?: string }[];
+ if (cachedClients.some((store) => store.directoryName === id && store.id !== id)) {
+ return false;
+ }
+
return true;
} catch {
return false;
diff --git a/packages/memory-storage/src/resource-clients/dataset.ts b/packages/memory-storage/src/resource-clients/dataset.ts
index a93c6a431..78cb8cd85 100644
--- a/packages/memory-storage/src/resource-clients/dataset.ts
+++ b/packages/memory-storage/src/resource-clients/dataset.ts
@@ -29,6 +29,12 @@ const LOCAL_ENTRY_NAME_DIGITS = 9;
export interface DatasetClientOptions {
id?: string;
name?: string;
+ /**
+ * The directory name to use on disk. When provided, takes precedence over `name` and `id`
+ * for the directory path. This allows alias-opened storages to have a directory name
+ * that differs from their metadata `name` (which is `undefined` for unnamed storages).
+ */
+ directoryName?: string;
baseStorageDirectory: string;
client: MemoryStorage;
}
@@ -38,6 +44,11 @@ export class DatasetClient<Data extends Dictionary = Dictionary>
implements storage.DatasetClient<Data>
{
name?: string;
+ /**
+ * The key used for directory naming and cache lookup. For named storages, this equals
+ * the name. For alias (unnamed) storages, this is the alias string. Falls back to id.
+ */
+ directoryName: string;
createdAt = new Date();
accessedAt = new Date();
modifiedAt = new Date();
@@ -50,7 +61,8 @@ export class DatasetClient<Data extends Dictionary = Dictionary>
constructor(options: DatasetClientOptions) {
super(options.id ?? randomUUID());
this.name = options.name;
- this.datasetDirectory = resolveWithinDirectory(options.baseStorageDirectory, this.name ?? this.id);
+ this.directoryName = options.directoryName ?? this.name ?? this.id;
+ this.datasetDirectory = resolveWithinDirectory(options.baseStorageDirectory, this.directoryName);
this.client = options.client;
}
diff --git a/packages/memory-storage/src/resource-clients/key-value-store.ts b/packages/memory-storage/src/resource-clients/key-value-store.ts
index 209035d4b..f11138c98 100644
--- a/packages/memory-storage/src/resource-clients/key-value-store.ts
+++ b/packages/memory-storage/src/resource-clients/key-value-store.ts
@@ -18,6 +18,12 @@ const DEFAULT_LOCAL_FILE_EXTENSION = 'bin';
export interface KeyValueStoreClientOptions {
name?: string;
id?: string;
+ /**
+ * The directory name to use on disk. When provided, takes precedence over `name` and `id`
+ * for the directory path. This allows alias-opened storages to have a directory name
+ * that differs from their metadata `name` (which is `undefined` for unnamed storages).
+ */
+ directoryName?: string;
baseStorageDirectory: string;
client: MemoryStorage;
}
@@ -32,6 +38,11 @@ export interface InternalKeyRecord {
export class KeyValueStoreClient extends BaseClient implements storage.KeyValueStoreClient {
name?: string;
+ /**
+ * The key used for directory naming and cache lookup. For named storages, this equals
+ * the name. For alias (unnamed) storages, this is the alias string. Falls back to id.
+ */
+ directoryName: string;
createdAt = new Date();
accessedAt = new Date();
modifiedAt = new Date();
@@ -43,7 +54,8 @@ export class KeyValueStoreClient extends BaseClient implements storage.KeyValueS
constructor(options: KeyValueStoreClientOptions) {
super(options.id ?? randomUUID());
this.name = options.name;
- this.keyValueStoreDirectory = resolveWithinDirectory(options.baseStorageDirectory, this.name ?? this.id);
+ this.directoryName = options.directoryName ?? this.name ?? this.id;
+ this.keyValueStoreDirectory = resolveWithinDirectory(options.baseStorageDirectory, this.directoryName);
this.client = options.client;
}
diff --git a/packages/memory-storage/src/resource-clients/request-queue.ts b/packages/memory-storage/src/resource-clients/request-queue.ts
index 80b85cde6..3699edbd5 100644
--- a/packages/memory-storage/src/resource-clients/request-queue.ts
+++ b/packages/memory-storage/src/resource-clients/request-queue.ts
@@ -35,6 +35,12 @@ const requestOptionsShape = s.object({
export interface RequestQueueClientOptions {
name?: string;
id?: string;
+ /**
+ * The directory name to use on disk. When provided, takes precedence over `name` and `id`
+ * for the directory path. This allows alias-opened storages to have a directory name
+ * that differs from their metadata `name` (which is `undefined` for unnamed storages).
+ */
+ directoryName?: string;
baseStorageDirectory: string;
client: MemoryStorage;
}
@@ -51,6 +57,11 @@ export interface InternalRequest {
export class RequestQueueClient extends BaseClient implements storage.RequestQueueClient {
name?: string;
+ /**
+ * The key used for directory naming and cache lookup. For named storages, this equals
+ * the name. For alias (unnamed) storages, this is the alias string. Falls back to id.
+ */
+ directoryName: string;
createdAt = new Date();
accessedAt = new Date();
modifiedAt = new Date();
@@ -66,7 +77,8 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQue
constructor(options: RequestQueueClientOptions) {
super(options.id ?? randomUUID());
this.name = options.name;
- this.requestQueueDirectory = resolveWithinDirectory(options.baseStorageDirectory, this.name ?? this.id);
+ this.directoryName = options.directoryName ?? this.name ?? this.id;
+ this.requestQueueDirectory = resolveWithinDirectory(options.baseStorageDirectory, this.directoryName);
this.client = options.client;
}
diff --git a/packages/types/src/storages.ts b/packages/types/src/storages.ts
index b992a5817..948af016b 100644
--- a/packages/types/src/storages.ts
+++ b/packages/types/src/storages.ts
@@ -297,36 +297,35 @@ export interface SetStatusMessageOptions {
}
/**
- * Identifies a storage by either its ID or its name, but not both.
- * If neither is provided, the default storage for the given type is used.
- */
-export type StorageIdentifier =
- | { id: string; name?: never }
- | { id?: never; name: string }
- | { id?: never; name?: never };
-
-/**
- * Options for opening a storage client. At most one of `id`, `name`, or `alias` may be provided.
- * `alias` identifies a run-scoped unnamed storage (e.g. `'__default__'` for the default storage).
+ * Identifies a storage by its ID, name, or alias. At most one may be provided.
*
- * TODO merge this into `StorageIdentifier` in https://github.com/apify/crawlee/issues/3074
+ * - `{ id }` — open a pre-existing storage by its unique ID.
+ * - `{ name }` — open or create a globally named storage (persists across runs).
+ * - `{ alias }` — open or create a run-scoped unnamed storage identified by this alias.
+ * The alias is used locally (e.g. as a directory name or cache key) but the storage
+ * itself has no persistent name. Use this for non-default unnamed storages.
+ * - `{}` / omitted — open the default storage.
*/
-export type StorageClientOpenOptions = StorageIdentifier | { id?: never; name?: never; alias: string };
+export type StorageIdentifier =
+ | { id: string; name?: never; alias?: never }
+ | { id?: never; name: string; alias?: never }
+ | { id?: never; name?: never; alias: string }
+ | { id?: never; name?: never; alias?: never };
/**
* Options for creating a dataset client via {@apilink StorageClient.createDatasetClient}.
*/
-export type CreateDatasetClientOptions = StorageClientOpenOptions;
+export type CreateDatasetClientOptions = StorageIdentifier;
/**
* Options for creating a key-value store client via {@apilink StorageClient.createKeyValueStoreClient}.
*/
-export type CreateKeyValueStoreClientOptions = StorageClientOpenOptions;
+export type CreateKeyValueStoreClientOptions = StorageIdentifier;
/**
* Options for creating a request queue client via {@apilink StorageClient.createRequestQueueClient}.
*/
-export type CreateRequestQueueClientOptions = StorageClientOpenOptions & {
+export type CreateRequestQueueClientOptions = StorageIdentifier & {
/**
* Client key for request locking.
* TODO: This is an Apify-platform concern and should eventually be pushed down
diff --git a/test/core/storages/storage_aliases.test.ts b/test/core/storages/storage_aliases.test.ts
new file mode 100644
index 000000000..919292f3a
--- /dev/null
+++ b/test/core/storages/storage_aliases.test.ts
@@ -0,0 +1,223 @@
+import { Dataset, KeyValueStore, RequestQueue } from '@crawlee/core';
+import { MemoryStorageEmulator } from '../../shared/MemoryStorageEmulator.js';
+
+const localStorageEmulator = new MemoryStorageEmulator();
+
+beforeEach(async () => {
+ await localStorageEmulator.init();
+});
+
+afterAll(async () => {
+ await localStorageEmulator.destroy();
+});
+
+describe('storage aliases', () => {
+ describe('Dataset.open with alias', () => {
+ test('should open a dataset with an alias', async () => {
+ const dataset = await Dataset.open({ alias: 'my-data' });
+ expect(dataset).toBeDefined();
+ expect(dataset.id).toBeDefined();
+ // Alias storages are unnamed
+ expect(dataset.name).toBeUndefined();
+ });
+
+ test('should return the same instance for the same alias', async () => {
+ const dataset1 = await Dataset.open({ alias: 'my-data' });
+ const dataset2 = await Dataset.open({ alias: 'my-data' });
+ expect(dataset1).toBe(dataset2);
+ });
+
+ test('should return different instances for different aliases', async () => {
+ const dataset1 = await Dataset.open({ alias: 'data-a' });
+ const dataset2 = await Dataset.open({ alias: 'data-b' });
+ expect(dataset1).not.toBe(dataset2);
+ expect(dataset1.id).not.toBe(dataset2.id);
+ });
+
+ test('alias storage should be independent from default storage', async () => {
+ const defaultDataset = await Dataset.open();
+ const aliasDataset = await Dataset.open({ alias: 'other' });
+ expect(defaultDataset).not.toBe(aliasDataset);
+ expect(defaultDataset.id).not.toBe(aliasDataset.id);
+ });
+
+ test('should store and retrieve data independently per alias', async () => {
+ const datasetA = await Dataset.open({ alias: 'store-a' });
+ const datasetB = await Dataset.open({ alias: 'store-b' });
+
+ await datasetA.pushData({ source: 'a' });
+ await datasetB.pushData({ source: 'b' });
+
+ await expect(datasetA.getData()).resolves.toMatchObject({ items: [{ source: 'a' }] });
+ await expect(datasetB.getData()).resolves.toMatchObject({ items: [{ source: 'b' }] });
+ });
+ });
+
+ describe('KeyValueStore.open with alias', () => {
+ test('should open a KVS with an alias', async () => {
+ const store = await KeyValueStore.open({ alias: 'my-store' });
+ expect(store).toBeDefined();
+ expect(store.id).toBeDefined();
+ expect(store.name).toBeUndefined();
+ });
+
+ test('should return the same instance for the same alias', async () => {
+ const store1 = await KeyValueStore.open({ alias: 'my-store' });
+ const store2 = await KeyValueStore.open({ alias: 'my-store' });
+ expect(store1).toBe(store2);
+ });
+
+ test('should store and retrieve values independently per alias', async () => {
+ const storeA = await KeyValueStore.open({ alias: 'kvs-a' });
+ const storeB = await KeyValueStore.open({ alias: 'kvs-b' });
+
+ await storeA.setValue('key', 'value-a');
+ await storeB.setValue('key', 'value-b');
+
+ await expect(storeA.getValue('key')).resolves.toBe('value-a');
+ await expect(storeB.getValue('key')).resolves.toBe('value-b');
+ });
+ });
+
+ describe('RequestQueue.open with alias', () => {
+ test('should return the same instance for the same alias', async () => {
+ const queue1 = await RequestQueue.open({ alias: 'my-queue' });
+ const queue2 = await RequestQueue.open({ alias: 'my-queue' });
+ expect(queue1).toBe(queue2);
+ });
+
+ test('should return different instances for different aliases', async () => {
+ const queue1 = await RequestQueue.open({ alias: 'queue-a' });
+ const queue2 = await RequestQueue.open({ alias: 'queue-b' });
+ expect(queue1).not.toBe(queue2);
+ expect(queue1.id).not.toBe(queue2.id);
+ });
+ });
+
+ describe('name/alias conflict detection', () => {
+ test('should throw when opening alias that conflicts with existing named dataset', async () => {
+ await Dataset.open({ name: 'shared-name' });
+ await expect(Dataset.open({ alias: 'shared-name' })).rejects.toThrow(
+ /Cannot open storage with alias "shared-name" because a named storage with the same identifier already exists/,
+ );
+ });
+
+ test('should throw when opening named dataset that conflicts with existing alias', async () => {
+ await Dataset.open({ alias: 'shared-name' });
+ await expect(Dataset.open({ name: 'shared-name' })).rejects.toThrow(
+ /Cannot open storage with name "shared-name" because an alias storage with the same identifier already exists\. If you meant to open the alias storage, use \{ alias: "shared-name" \} instead\./,
+ );
+ });
+
+ test('should throw when opening alias that conflicts with existing named KVS', async () => {
+ await KeyValueStore.open({ name: 'shared-kvs' });
+ await expect(KeyValueStore.open({ alias: 'shared-kvs' })).rejects.toThrow(
+ /Cannot open storage with alias "shared-kvs" because a named storage with the same identifier already exists/,
+ );
+ });
+
+ test('should not conflict across different storage types', async () => {
+ // A dataset name and a KVS alias with the same string should not conflict
+ await Dataset.open({ name: 'cross-type' });
+ const store = await KeyValueStore.open({ alias: 'cross-type' });
+ expect(store).toBeDefined();
+ });
+ });
+
+ describe('string identifier vs alias conflict', () => {
+ test('should throw when opening a string identifier that matches an existing alias', async () => {
+ await Dataset.open({ alias: 'asdf' });
+ // 'asdf' as a bare string resolves to { name: 'asdf' }, which should conflict with alias 'asdf'
+ await expect(Dataset.open('asdf')).rejects.toThrow(
+ /Cannot open storage with name "asdf" because an alias storage with the same identifier already exists\. If you meant to open the alias storage, use \{ alias: "asdf" \} instead\./,
+ );
+ });
+
+ test('should throw when opening an alias that matches an existing string-opened storage', async () => {
+ await Dataset.open('asdf');
+ await expect(Dataset.open({ alias: 'asdf' })).rejects.toThrow(
+ /Cannot open storage with alias "asdf" because a named storage with the same identifier already exists/,
+ );
+ });
+ });
+
+ describe('string identifier vs alias conflict (persistent storage)', () => {
+ const persistentEmulator = new MemoryStorageEmulator();
+
+ beforeEach(async () => {
+ await persistentEmulator.init({ persistStorage: true });
+ });
+
+ afterAll(async () => {
+ await persistentEmulator.destroy();
+ });
+
+ test('should throw when opening a string identifier that matches an existing alias on disk', async () => {
+ await Dataset.open({ alias: 'on-disk' });
+ // With persistence, the directory 'on-disk' exists on disk. storageExists() should
+ // not be fooled into treating the string as an ID.
+ await expect(Dataset.open('on-disk')).rejects.toThrow(
+ /Cannot open storage with name "on-disk" because an alias storage with the same identifier already exists\. If you meant to open the alias storage, use \{ alias: "on-disk" \} instead\./,
+ );
+ });
+ });
+
+ describe('resolveStorageIdentifier', () => {
+ test('null identifier opens default storage', async () => {
+ const dataset1 = await Dataset.open(null);
+ const dataset2 = await Dataset.open();
+ expect(dataset1).toBe(dataset2);
+ });
+
+ test('undefined identifier opens default storage', async () => {
+ const dataset1 = await Dataset.open(undefined);
+ const dataset2 = await Dataset.open();
+ expect(dataset1).toBe(dataset2);
+ });
+
+ test('empty object opens default storage', async () => {
+ const dataset1 = await Dataset.open({});
+ const dataset2 = await Dataset.open();
+ expect(dataset1).toBe(dataset2);
+ });
+
+ test('string identifier opens named storage', async () => {
+ const dataset = await Dataset.open('test-named');
+ expect(dataset.name).toBe('test-named');
+ });
+
+ test('{ name } identifier opens named storage', async () => {
+ const dataset = await Dataset.open({ name: 'test-named-obj' });
+ expect(dataset.name).toBe('test-named-obj');
+ });
+
+ test('{ alias } identifier opens alias storage', async () => {
+ const dataset = await Dataset.open({ alias: 'test-alias' });
+ expect(dataset.name).toBeUndefined();
+ });
+ });
+
+ describe('drop with alias', () => {
+ test('should be able to drop an aliased dataset and re-open it', async () => {
+ const dataset1 = await Dataset.open({ alias: 'droppable' });
+ await dataset1.pushData({ foo: 'bar' });
+ await dataset1.drop();
+
+ const dataset2 = await Dataset.open({ alias: 'droppable' });
+ expect(dataset2).not.toBe(dataset1);
+ await expect(dataset2.getData()).resolves.toMatchObject({ items: [] });
+ });
+ });
+
+ describe('concurrent alias opens', () => {
+ test('should handle concurrent opens of the same alias', async () => {
+ const [d1, d2, d3] = await Promise.all([
+ Dataset.open({ alias: 'concurrent' }),
+ Dataset.open({ alias: 'concurrent' }),
+ Dataset.open({ alias: 'concurrent' }),
+ ]);
+ expect(d1).toBe(d2);
+ expect(d2).toBe(d3);
+ });
+ });
+});