Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
refactor!: Split MemoryStorage into FileSystemStorageClient and MemoryStorageClient (#3740)
Jan Buchar2f0a60d52db8
diff --git a/REBASE-DEFERRED.md b/REBASE-DEFERRED.mdindex e9016a01f..41ba743b5 100644--- a/REBASE-DEFERRED.md+++ b/REBASE-DEFERRED.md@@ -355,3 +355,50 @@ rename + rewrite, not a simple edit). Git's merge left the old test's openingtwo lines behind as an unconflicting fragment ahead of the conflict region;the renamed test's full new body already landed earlier in the file(confirmed via grep, `line 61`). Deleted the orphaned fragment.++**Commit `a9b972294` (Split MemoryStorage into FileSystemStorageClient and+MemoryStorageClient) — ported the path-traversal hardening into the new+`@crawlee/fs-storage` package**: this commit splits the old+`@crawlee/memory-storage` into a pure in-memory client (stays in+`memory-storage`, drops all filesystem code) and a new `@crawlee/fs-storage`+package (the file-system-backed implementation, entirely new files, not+conflicted). The new `fs-storage` resource clients+(`dataset.ts`/`key-value-store.ts`/`request-queue.ts`) build their storage+directory as `resolve(baseStorageDirectory, directoryName)` using plain+`node:path` `resolve()` — this is the exact path-traversal vulnerability+master's `resolveWithinDirectory()` helper (commit `a04c29766+fix(memory-storage): prevent storage names from escaping the storage+directory (#3715)`) fixed for the old combined package. Since `fs-storage` is+a wholesale new copy of that same pre-fix code (not derived from+`memory-storage`'s post-fix version), the vulnerability was reintroduced.+Ported `resolveWithinDirectory` into `packages/fs-storage/src/utils.ts` and+switched all three resource clients' directory-construction call to use it+(left the `rm(resolve(this.xDirectory, entry))` calls alone — those resolve+internally-generated entry names, not user-controlled names/keys, matching+the scope of the original fix). Resolved the conflicts in the `memory-storage`+copies of these same three files by taking theirs' side throughout (drop+`directoryName`/`resolveWithinDirectory`/filesystem imports in favor of the+new `cacheKey`-only in-memory identity) — `memory-storage` no longer touches+the filesystem at all post-split, so the hardening there is not applicable+(it now lives solely in `fs-storage`).++Note: the original `a04c29766` fix's description also mentions hardening+key-value-store record keys (`setRecord({ key })`), not just storage names.+I only found and fixed the storage-name/directoryName call site in the new+`fs-storage` package — worth a follow-up check once the rebase is done to see+whether record-key path construction (likely inside+`fs-storage/src/fs/key-value-store/*.ts`) needs the same treatment.++**Follow-up on the above, resolved immediately**: checked+`packages/fs-storage/src/fs/key-value-store/fs.ts` — it already used+`resolveWithinDirectory` for its main `update()` path (carried over cleanly+from an earlier point in this rebase), but its `get()` fallback (the+no-file-extension retry path) still called bare `resolve(this.storeDirectory,+this.rawRecord.key)` on a record key — worse, `resolve` wasn't even imported+in that file anymore, so this line would have been a compile error. Fixed to+use `resolveWithinDirectory`, matching master. Checked the sibling+`fs/dataset/fs.ts` and `fs/request-queue/fs.ts` for the same pattern — both+build their file path from internally-generated, non-user-controlled IDs+(sequential entity index, hashed request ID), which is exactly what master's+original fix (`a04c29766`) called out as already safe, so left them as+plain `resolve()`.diff --git a/docs/guides/request_storage.mdx b/docs/guides/request_storage.mdxindex 11aed2831..49ea41359 100644--- a/docs/guides/request_storage.mdx+++ b/docs/guides/request_storage.mdx@@ -26,7 +26,7 @@ Each Crawlee project run is associated with a **default request queue**. TypicalIn Crawlee, the request queue is represented by the <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink> class.-The request queue is managed by <ApiLink to="memory-storage/class/MemoryStorage">`MemoryStorage`</ApiLink> class and its data is stored in memory, while also being off-loaded to the local directory specified by the `CRAWLEE_STORAGE_DIR` environment variable as follows:+By default, the request queue is managed by the <ApiLink to="fs-storage/class/FileSystemStorageClient">`FileSystemStorageClient`</ApiLink> class and its data is stored in the local directory specified by the `CRAWLEE_STORAGE_DIR` environment variable as follows:```text{CRAWLEE_STORAGE_DIR}/request_queues/{QUEUE_ID}/entries.jsondiff --git a/docs/guides/result_storage.mdx b/docs/guides/result_storage.mdxindex eff313aef..583f613c6 100644--- a/docs/guides/result_storage.mdx+++ b/docs/guides/result_storage.mdx@@ -8,7 +8,7 @@ import ApiLink from '@site/src/components/ApiLink';Crawlee has several result storage types that are useful for specific tasks. The data is stored on a local disk to the directory defined by the `CRAWLEE_STORAGE_DIR` environment variable. If this variable is not defined, by default Crawlee sets `CRAWLEE_STORAGE_DIR` to `./storage` in the current working directory.-Crawlee storage is managed by <ApiLink to="memory-storage/class/MemoryStorage">`MemoryStorage`</ApiLink> class. During the crawler run all information is stored in memory, while also being off-loaded to the local files in respective storage type folders.+By default, Crawlee storage is managed by the <ApiLink to="fs-storage/class/FileSystemStorageClient">`FileSystemStorageClient`</ApiLink> class, which stores all information as local files in the respective storage type folders.## Key-value storediff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.mdindex 543284cb4..ad0258db4 100644--- a/docs/upgrading/upgrading_v4.md+++ b/docs/upgrading/upgrading_v4.md@@ -431,14 +431,14 @@ The new `ServiceLocator` supports per-crawler service isolation, allowing you to```typescriptimport { BasicCrawler, Configuration, LocalEventManager } from 'crawlee';-import { MemoryStorage } from '@crawlee/memory-storage';+import { MemoryStorageClient } from '@crawlee/memory-storage';const crawler = new BasicCrawler({requestHandler: async ({ request, log }) => {log.info(`Processing ${request.url}`);},configuration: new Configuration({ headless: false }),- storageClient: new MemoryStorage(),+ storageClient: new MemoryStorageClient(),eventManager: LocalEventManager.fromConfig(),});@@ -451,10 +451,10 @@ For most use cases, the global `serviceLocator` singleton works well:```typescriptimport { serviceLocator, BasicCrawler } from 'crawlee';-import { MemoryStorage } from '@crawlee/memory-storage';+import { MemoryStorageClient } from '@crawlee/memory-storage';// Configure global services (optional)-serviceLocator.setStorageClient(new MemoryStorage());+serviceLocator.setStorageClient(new MemoryStorageClient());// All crawlers will use the global service locator by defaultconst crawler = new BasicCrawler({@@ -643,6 +643,52 @@ If you implemented a custom `StorageClient`, you need to:2. Replace the six getter methods (`dataset`, `datasets`, `keyValueStore`, `keyValueStores`, `requestQueue`, `requestQueues`) with three async factory methods (`createDatasetClient`, `createKeyValueStoreClient`, `createRequestQueueClient`). Each factory should handle both opening an existing storage and creating a new one.3. Apply the sub-client renames listed above (`get` → `getMetadata`, `delete` → `drop`, etc.) and implement the new `purge()` method.+## `MemoryStorage` split into `FileSystemStorageClient` and `MemoryStorageClient`++In v3, the single `MemoryStorage` class from `@crawlee/memory-storage` did double duty: it kept everything in memory *and*, by default, mirrored it to disk (toggled via the `persistStorage` option / `CRAWLEE_PERSIST_STORAGE` environment variable). In v4 these two responsibilities are split into two independent classes, and the default storage client now persists to disk.++- **`FileSystemStorageClient`** (new, in the new `@crawlee/fs-storage` package) — always persists storage to the local directory (`CRAWLEE_STORAGE_DIR`, default `./storage`). This is what you get implicitly when you don't configure a storage client, and it is the behavior the old `MemoryStorage` had with its default `persistStorage: true`.+- **`MemoryStorageClient`** (the renamed `MemoryStorage`, still in `@crawlee/memory-storage`) — now keeps everything purely in memory and **never touches the disk**. This matches the old `MemoryStorage` with `persistStorage: false`.++Both classes are re-exported from the `crawlee` meta-package.++### The default storage client now persists to disk++Which client backs the implicit default is decided by `Configuration.persistStorage` (still controllable via the `CRAWLEE_PERSIST_STORAGE` environment variable): `true` (the default) selects `FileSystemStorageClient`, `false` selects `MemoryStorageClient`. If you relied on the default and never set `persistStorage`, your storage is persisted to disk exactly as before — no change.++### `MemoryStorage` is renamed and is now memory-only++If you constructed the storage client explicitly, two things changed:++1. **The class is renamed** `MemoryStorage` → `MemoryStorageClient`.+2. **It no longer writes to disk.** A bare `new MemoryStorage()` in v3 persisted to disk by default; `new MemoryStorageClient()` in v4 does not. If you want persistence, use `FileSystemStorageClient` instead.++**Before:**+```typescript+import { MemoryStorage } from '@crawlee/memory-storage';++// Persisted to disk by default in v3.+const storageClient = new MemoryStorage();+```++**After:**+```typescript+import { FileSystemStorageClient } from '@crawlee/fs-storage';+import { MemoryStorageClient } from '@crawlee/memory-storage';++// Persists to disk (the old default behavior):+const storageClient = new FileSystemStorageClient();++// Or keep everything in memory only (the old `persistStorage: false`):+const inMemory = new MemoryStorageClient();+```++The `localDataDirectory`, `persistStorage`, and `writeMetadata` options are still accepted by `MemoryStorageClient` for source compatibility, but they are ignored — in-memory storage has nowhere to write. `FileSystemStorageClient` honors `localDataDirectory` and `writeMetadata`; it always persists, so it has no `persistStorage` option.++### No request lock expiry in `MemoryStorageClient`++Because the in-memory queue lives entirely within a single process and is never shared with another consumer, `MemoryStorageClient`'s request queue no longer uses an expiring, cross-process lock. A fetched request simply stays *in progress* until it is handled or reclaimed; it never becomes fetchable again on its own after a timeout. `setExpectedRequestProcessingTimeSecs()` is therefore a no-op for in-memory storage. (Disk-backed `FileSystemStorageClient` keeps the lock-with-expiry behavior.)+## Multiple crawler instances use separate default request queuesIn v3, every `BasicCrawler` (or subclass) that didn't receive an explicit `requestQueue` option would open the same default request queue. If you created two crawlers in the same process, they would silently share a queue — leading to request collisions and hard-to-debug deduplication issues.diff --git a/packages/core/package.json b/packages/core/package.jsonindex 0bef02294..e5de74aec 100644--- a/packages/core/package.json+++ b/packages/core/package.json@@ -53,6 +53,7 @@"@apify/pseudo_url": "^2.0.59","@apify/timeout": "^0.3.2","@apify/utilities": "^2.15.5",+ "@crawlee/fs-storage": "workspace:*","@crawlee/memory-storage": "workspace:*","@crawlee/types": "workspace:*","@crawlee/utils": "workspace:*",diff --git a/packages/core/src/service_locator.ts b/packages/core/src/service_locator.tsindex 8ffb98cac..8cecd5f0d 100644--- a/packages/core/src/service_locator.ts+++ b/packages/core/src/service_locator.ts@@ -1,6 +1,7 @@import { AsyncLocalStorage } from 'node:async_hooks';-import { MemoryStorage } from '@crawlee/memory-storage';+import { FileSystemStorageClient } from '@crawlee/fs-storage';+import { MemoryStorageClient } from '@crawlee/memory-storage';import type { StorageClient } from '@crawlee/types';import log from '@apify/log';@@ -44,7 +45,8 @@ interface ServiceLocatorInterface {/*** Get the storage client.- * Creates a default MemoryStorage instance if none has been set.+ * Creates a default storage client if none has been set — `FileSystemStorageClient` when+ * `persistStorage` is enabled (the default), `MemoryStorageClient` otherwise.*/getStorageClient(): StorageClient;@@ -110,12 +112,12 @@ interface ServiceLocatorInterface {* **2. Per-crawler services (recommended for isolation):*** ```typescript* import { BasicCrawler, Configuration, LocalEventManager } from 'crawlee';- * import { MemoryStorage } from '@crawlee/memory-storage';+ * import { MemoryStorageClient } from '@crawlee/memory-storage';** const crawler = new BasicCrawler({* requestHandler: async ({ request }) => { ... },* configuration: new Configuration({ ... }), // custom config- * storageClient: new MemoryStorage(), // custom storage+ * storageClient: new MemoryStorageClient(), // custom storage* eventManager: LocalEventManager.fromConfig(), // custom events* });* // Crawler has its own isolated ServiceLocator instance@@ -206,7 +208,10 @@ export class ServiceLocator implements ServiceLocatorInterface {getStorageClient(): StorageClient {if (!this.storageClient) {- this.getLogger().debug('No storage client set, implicitly creating and using default MemoryStorage.');+ this.getLogger().debug(+ 'No storage client set, implicitly creating and using the default storage client ' ++ '(FileSystemStorageClient when persistStorage is enabled, MemoryStorageClient otherwise).',+ );if (!this.configuration) {this.getLogger().warning('Implicit creation of storage client will implicitly set configuration as side effect. ' +@@ -214,10 +219,13 @@ export class ServiceLocator implements ServiceLocatorInterface {);}const config = this.getConfiguration();- this.storageClient = new MemoryStorage({- persistStorage: config.persistStorage,- logger: this.getLogger().child({ prefix: 'MemoryStorage' }),- });+ this.storageClient = config.persistStorage+ ? new FileSystemStorageClient({+ logger: this.getLogger().child({ prefix: 'FileSystemStorageClient' }),+ })+ : new MemoryStorageClient({+ logger: this.getLogger().child({ prefix: 'MemoryStorageClient' }),+ });}return this.storageClient;}diff --git a/packages/core/test/core/service_locator.test.ts b/packages/core/test/core/service_locator.test.tsindex 4429a1ac6..8efee82e9 100644--- a/packages/core/test/core/service_locator.test.ts+++ b/packages/core/test/core/service_locator.test.ts@@ -7,7 +7,8 @@ import {ServiceLocator,serviceLocator,} from '@crawlee/core';-import { MemoryStorage } from '@crawlee/memory-storage';+import { FileSystemStorageClient } from '@crawlee/fs-storage';+import { MemoryStorageClient } from '@crawlee/memory-storage';function makeMockLogger(overrides: Partial<CrawleeLogger> = {}): CrawleeLogger {const logger: CrawleeLogger = {@@ -132,11 +133,11 @@ describe('ServiceLocator', () => {describe('StorageClient', () => {test('default storage client', () => {const defaultStorageClient = serviceLocator.getStorageClient();- expect(defaultStorageClient).toBeInstanceOf(MemoryStorage);+ expect(defaultStorageClient).toBeInstanceOf(FileSystemStorageClient);});test('custom storage client', () => {- const customStorageClient = new MemoryStorage();+ const customStorageClient = new MemoryStorageClient();serviceLocator.setStorageClient(customStorageClient);const storageClient = serviceLocator.getStorageClient();@@ -144,10 +145,10 @@ describe('ServiceLocator', () => {});test('storage client overwrite not possible', () => {- const customStorageClient = new MemoryStorage();+ const customStorageClient = new MemoryStorageClient();serviceLocator.setStorageClient(customStorageClient);- const anotherCustomStorageClient = new MemoryStorage();+ const anotherCustomStorageClient = new MemoryStorageClient();expect(() => {serviceLocator.setStorageClient(anotherCustomStorageClient);@@ -158,7 +159,7 @@ describe('ServiceLocator', () => {// Retrieve storage client firstserviceLocator.getStorageClient();- const customStorageClient = new MemoryStorage();+ const customStorageClient = new MemoryStorageClient();expect(() => {serviceLocator.setStorageClient(customStorageClient);@@ -238,7 +239,7 @@ describe('ServiceLocator', () => {persistStateIntervalMillis: 1000,systemInfoIntervalMillis: 1000,});- const customStorageClient = new MemoryStorage();+ const customStorageClient = new MemoryStorageClient();serviceLocator.setConfiguration(customConfig);serviceLocator.setEventManager(customEventManager);@@ -287,7 +288,7 @@ describe('ServiceLocator', () => {});test('setting same storage client instance is allowed', () => {- const storageClient = new MemoryStorage();+ const storageClient = new MemoryStorageClient();serviceLocator.setStorageClient(storageClient);serviceLocator.getStorageClient();@@ -342,7 +343,7 @@ describe('ServiceLocator', () => {describe('Per-crawler ServiceLocator', () => {test('creating separate service locator for crawler', () => {const crawlerConfig = new Configuration({ headless: false });- const crawlerStorage = new MemoryStorage();+ const crawlerStorage = new MemoryStorageClient();const crawlerEvents = new LocalEventManager({persistStateIntervalMillis: 1000,systemInfoIntervalMillis: 1000,diff --git a/packages/core/test/request-queue/adding-the-same-request-should-not-call-the-api.test.ts b/packages/core/test/request-queue/adding-the-same-request-should-not-call-the-api.test.tsindex 831bf3c46..c7c07bb12 100644--- a/packages/core/test/request-queue/adding-the-same-request-should-not-call-the-api.test.ts+++ b/packages/core/test/request-queue/adding-the-same-request-should-not-call-the-api.test.ts@@ -1,11 +1,11 @@-import { MemoryStorage } from '@crawlee/memory-storage';+import { MemoryStorageClient } from '@crawlee/memory-storage';import type { RequestQueueClient } from '@crawlee/types';import { RequestQueue, serviceLocator } from 'crawlee';let rqClient: RequestQueueClient;beforeEach(async () => {- const storage = new MemoryStorage({ persistStorage: false, writeMetadata: false });+ const storage = new MemoryStorageClient();serviceLocator.setStorageClient(storage);rqClient = await storage.createRequestQueueClient({ name: 'test-request-queue-not-called-on-cached-request' });});diff --git a/packages/core/test/request-queue/request-queue.test.ts b/packages/core/test/request-queue/request-queue.test.tsindex a14e8040e..8d31608fb 100644--- a/packages/core/test/request-queue/request-queue.test.ts+++ b/packages/core/test/request-queue/request-queue.test.ts@@ -1,8 +1,8 @@-import { MemoryStorage } from '@crawlee/memory-storage';+import { MemoryStorageClient } from '@crawlee/memory-storage';import { RequestQueue } from 'crawlee';import type { MockInstance } from 'vitest';-const storage = new MemoryStorage({ persistStorage: false, writeMetadata: false });+const storage = new MemoryStorageClient();async function makeQueue(name: string, numOfRequestsToAdd = 0) {const rqClient = await storage.createRequestQueueClient({ name });diff --git a/packages/core/test/storages/open-storage-with-different-client-should-be-respected.test.ts b/packages/core/test/storages/open-storage-with-different-client-should-be-respected.test.tsindex 3c479394c..5d331bb04 100644--- a/packages/core/test/storages/open-storage-with-different-client-should-be-respected.test.ts+++ b/packages/core/test/storages/open-storage-with-different-client-should-be-respected.test.ts@@ -1,10 +1,10 @@-import { MemoryStorage } from '@crawlee/memory-storage';+import { MemoryStorageClient } from '@crawlee/memory-storage';import { RequestQueue, serviceLocator } from 'crawlee';-let newClient: MemoryStorage;+let newClient: MemoryStorageClient;beforeEach(() => {- newClient = new MemoryStorage({ persistStorage: false, writeMetadata: false });+ newClient = new MemoryStorageClient();serviceLocator.setStorageClient(newClient);});@@ -12,13 +12,13 @@ describe('Opening a storage with a different storage client should be respected'test('opening a RequestQueue with default client from Configuration', async () => {const queue = await RequestQueue.open({ name: 'test-rq-open-client-from-config' });- // The sub-client should have been created by newClient (MemoryStorage),+ // The sub-client should have been created by newClient (MemoryStorageClient),// so its internal `client` field should reference newClient.expect((queue.client as any).client).toBe(newClient);});test('opening a RequestQueue with a different client', async () => {- const thirdClient = new MemoryStorage({ persistStorage: false, writeMetadata: false });+ const thirdClient = new MemoryStorageClient();// @ts-expect-error Using this to ensure the test/impl worksthirdClient._name = 'third-client';diff --git a/packages/crawlee/package.json b/packages/crawlee/package.jsonindex 66e52eaa2..52428520b 100644--- a/packages/crawlee/package.json+++ b/packages/crawlee/package.json@@ -54,9 +54,11 @@"@crawlee/cheerio": "workspace:*","@crawlee/cli": "workspace:*","@crawlee/core": "workspace:*",+ "@crawlee/fs-storage": "workspace:*","@crawlee/http": "workspace:*","@crawlee/jsdom": "workspace:*","@crawlee/linkedom": "workspace:*",+ "@crawlee/memory-storage": "workspace:*","@crawlee/playwright": "workspace:*","@crawlee/puppeteer": "workspace:*","@crawlee/utils": "workspace:*",diff --git a/packages/crawlee/src/index.ts b/packages/crawlee/src/index.tsindex f96c34307..ef2827e0f 100644--- a/packages/crawlee/src/index.ts+++ b/packages/crawlee/src/index.ts@@ -15,6 +15,7 @@ export * from '@crawlee/puppeteer';export * from '@crawlee/playwright';export * from '@crawlee/browser-pool';export * from '@crawlee/memory-storage';+export * from '@crawlee/fs-storage';export const utils = {puppeteer: puppeteerUtils,diff --git a/packages/fs-storage/.npmignore b/packages/fs-storage/.npmignorenew file mode 100644index 000000000..d18b05a66--- /dev/null+++ b/packages/fs-storage/.npmignore@@ -0,0 +1,5 @@+node_modules+src+test+coverage+tsconfig.*diff --git a/packages/fs-storage/package.json b/packages/fs-storage/package.jsonnew file mode 100644index 000000000..ef6c35e52--- /dev/null+++ b/packages/fs-storage/package.json@@ -0,0 +1,55 @@+{+ "name": "@crawlee/fs-storage",+ "version": "4.0.0",+ "description": "A file-system storage implementation of the Apify API",+ "engines": {+ "node": ">=22.0.0"+ },+ "type": "module",+ "exports": {+ ".": "./dist/index.js",+ "./package.json": "./package.json"+ },+ "keywords": [+ "apify",+ "api",+ "filesystem"+ ],+ "author": {+ "name": "Apify",+ "email": "[email protected]",+ "url": "https://apify.com"+ },+ "contributors": [+ "Vlad Frangu <[email protected]>"+ ],+ "license": "Apache-2.0",+ "repository": {+ "type": "git",+ "url": "git+https://github.com/apify/crawlee"+ },+ "bugs": {+ "url": "https://github.com/apify/crawlee/issues"+ },+ "homepage": "https://crawlee.dev",+ "scripts": {+ "build": "pnpm clean && pnpm compile && pnpm copy",+ "clean": "rimraf ./dist",+ "compile": "tsc -p tsconfig.build.json",+ "copy": "tsx ../../scripts/copy.ts"+ },+ "publishConfig": {+ "access": "public"+ },+ "dependencies": {+ "@crawlee/types": "workspace:*",+ "@sapphire/async-queue": "^1.5.5",+ "@sapphire/shapeshift": "^4.0.0",+ "content-type": "^1.0.5",+ "fs-extra": "^11.3.0",+ "json5": "^2.2.3",+ "mime-types": "^3.0.1",+ "proper-lockfile": "^4.1.2",+ "tslib": "^2.8.1"+ }+}diff --git a/packages/memory-storage/src/background-handler/fs-utils.ts b/packages/fs-storage/src/background-handler/fs-utils.tssimilarity index 100%rename from packages/memory-storage/src/background-handler/fs-utils.tsrename to packages/fs-storage/src/background-handler/fs-utils.tsdiff --git a/packages/memory-storage/src/background-handler/index.ts b/packages/fs-storage/src/background-handler/index.tssimilarity index 90%rename from packages/memory-storage/src/background-handler/index.tsrename to packages/fs-storage/src/background-handler/index.tsindex 50417f56c..007a98e03 100644--- a/packages/memory-storage/src/background-handler/index.ts+++ b/packages/fs-storage/src/background-handler/index.ts@@ -7,7 +7,7 @@ import { handleMessage } from './fs-utils.js';/*** A map of promises that are created when a background task is scheduled.- * This is used in MemoryStorage#teardown to wait for all tasks to finish executing before exiting the process.+ * This is used in FileSystemStorageClient#teardown to wait for all tasks to finish executing before exiting the process.* @internal*/export const promiseMap: Map<diff --git a/packages/fs-storage/src/body-parser.ts b/packages/fs-storage/src/body-parser.tsnew file mode 100644index 000000000..92c6ce172--- /dev/null+++ b/packages/fs-storage/src/body-parser.ts@@ -0,0 +1,62 @@+import contentTypeParser from 'content-type';+import JSON5 from 'json5';++const CONTENT_TYPE_JSON = 'application/json';+const STRINGIFIABLE_CONTENT_TYPE_RXS = [new RegExp(`^${CONTENT_TYPE_JSON}$`, 'i'), /^application\/.*xml$/i, /^text\//i];++/**+ * Parses a Buffer or ArrayBuffer using the provided content type header.+ *+ * - application/json is returned as a parsed object.+ * - application/*xml and text/* are returned as strings.+ * - everything else is returned as original body.+ *+ * If the header includes a charset, the body will be stringified only+ * if the charset represents a known encoding to Node.js or Browser.+ */+export function maybeParseBody(+ body: Buffer | ArrayBuffer,+ contentTypeHeader: string,+): string | Buffer | ArrayBuffer | Record<string, unknown> {+ let contentType: string;+ let charset: BufferEncoding;+ try {+ const result = contentTypeParser.parse(contentTypeHeader);+ contentType = result.type;+ charset = result.parameters.charset as BufferEncoding;+ } catch {+ // can't parse, keep original body+ return body;+ }++ // If we can't successfully parse it, we return+ // the original buffer rather than a mangled string.+ if (!areDataStringifiable(contentType, charset)) return body;+ const dataString = isomorphicBufferToString(body, charset);++ return contentType === CONTENT_TYPE_JSON ? JSON5.parse(dataString) : dataString;+}++function isomorphicBufferToString(buffer: Buffer | ArrayBuffer, encoding: BufferEncoding): string {+ if (buffer.constructor.name !== ArrayBuffer.name) {+ return buffer.toString(encoding);+ }++ // Browser decoding only works with UTF-8.+ const utf8decoder = new TextDecoder();+ return utf8decoder.decode(new Uint8Array(buffer));+}++function isCharsetStringifiable(charset: string): charset is BufferEncoding {+ if (!charset) return true; // hope that it's utf-8+ return Buffer.isEncoding(charset);+}++function isContentTypeStringifiable(contentType: string): boolean {+ if (!contentType) return false; // keep buffer+ return STRINGIFIABLE_CONTENT_TYPE_RXS.some((rx) => rx.test(contentType));+}++function areDataStringifiable(contentType: string, charset: string): boolean {+ return isContentTypeStringifiable(contentType) && isCharsetStringifiable(charset);+}diff --git a/packages/memory-storage/src/cache-helpers.ts b/packages/fs-storage/src/cache-helpers.tssimilarity index 89%rename from packages/memory-storage/src/cache-helpers.tsrename to packages/fs-storage/src/cache-helpers.tsindex c0076b6d9..d4e3cfc51 100644--- a/packages/memory-storage/src/cache-helpers.ts+++ b/packages/fs-storage/src/cache-helpers.ts@@ -8,11 +8,11 @@ import mimeTypes from 'mime-types';import { DatasetFileSystemEntry } from './fs/dataset/fs.js';import { KeyValueFileSystemEntry } from './fs/key-value-store/fs.js';import { RequestQueueFileSystemEntry } from './fs/request-queue/fs.js';-import { type MemoryStorage } from './memory-storage.js';+import { type FileSystemStorageClient } from './file-system-storage.js';const uuidRegex = /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/i;-export async function findOrCacheDatasetByPossibleId(client: MemoryStorage, entryNameOrId: string) {+export async function findOrCacheDatasetByPossibleId(client: FileSystemStorageClient, entryNameOrId: string) {// First check memory cache — match by id, name, or directoryName (which covers alias lookups)const found = client.datasetClientCache.find((store) =>@@ -106,7 +106,6 @@ export async function findOrCacheDatasetByPossibleId(client: MemoryStorage, entrconst entry = new DatasetFileSystemEntry({storeDirectory: datasetDir,entityId: entryId,- persistStorage: true,});// eslint-disable-next-line dot-notation@@ -118,7 +117,7 @@ export async function findOrCacheDatasetByPossibleId(client: MemoryStorage, entrreturn newClient;}-export async function findOrCacheKeyValueStoreByPossibleId(client: MemoryStorage, entryNameOrId: string) {+export async function findOrCacheKeyValueStoreByPossibleId(client: FileSystemStorageClient, entryNameOrId: string) {// First check memory cache — match by id, name, or directoryName (which covers alias lookups)const found = client.keyValueStoreCache.find((store) =>@@ -253,7 +252,6 @@ export async function findOrCacheKeyValueStoreByPossibleId(client: MemoryStoragefor (const [key, record] of internalRecords) {// We create a file system entry instead of possibly making an in-memory one to allow the pre-included data to be used on demandconst entry = new KeyValueFileSystemEntry({- persistStorage: true,storeDirectory: keyValueStoreDir,writeMetadata: hasSeenMetadataForEntry,logger: client.logger,@@ -275,7 +273,7 @@ export async function findOrCacheKeyValueStoreByPossibleId(client: MemoryStoragereturn newClient;}-export async function findRequestQueueByPossibleId(client: MemoryStorage, entryNameOrId: string) {+export async function findRequestQueueByPossibleId(client: FileSystemStorageClient, entryNameOrId: string) {// First check memory cache — match by id, name, or directoryName (which covers alias lookups)const found = client.requestQueueCache.find((store) =>@@ -305,11 +303,16 @@ export async function findRequestQueueByPossibleId(client: MemoryStorage, entryNlet createdAt = new Date();let accessedAt = new Date();let modifiedAt = new Date();- let pendingRequestCount = 0;- let handledRequestCount = 0;const entries = new Set<string>();let forefrontRequestIds: string[] = [];+ // The request counts are derived from the request files actually present on disk rather than read+ // from the metadata file: metadata is only persisted when `writeMetadata` is enabled (off by+ // default), whereas request files are always persisted. Trusting the metadata counts would reset+ // them to 0 on reload whenever `writeMetadata` is off, even though the requests survive on disk.+ let pendingRequestCount = 0;+ let handledRequestCount = 0;+for await (const entry of directoryEntries) {if (entry.isFile()) {switch (entry.name) {@@ -325,8 +328,6 @@ export async function findRequestQueueByPossibleId(client: MemoryStorage, entryNcreatedAt = new Date(metadata.createdAt);accessedAt = new Date(metadata.accessedAt);modifiedAt = new Date(metadata.modifiedAt);- pendingRequestCount = metadata.pendingRequestCount;- handledRequestCount = metadata.handledRequestCount;forefrontRequestIds = (metadata as any)?.forefrontRequestIds ?? [];break;@@ -342,9 +343,16 @@ export async function findRequestQueueByPossibleId(client: MemoryStorage, entryNtry {// Try parsing the file to ensure it's even valid to begin withconst fileContent = await readFile(resolve(requestQueueDir, entry.name), 'utf8');- JSON.parse(fileContent);+ const parsed = JSON.parse(fileContent) as { orderNo?: number | null };entries.add(entryName);++ // A handled request has `orderNo === null`; anything else is still pending.+ if (parsed.orderNo === null) {+ handledRequestCount += 1;+ } else {+ pendingRequestCount += 1;+ }} catch {client.logger?.warning(`Request queue entry "${entry.name}" for store ${entryNameOrId} has invalid JSON content and will be ignored from the store.`,@@ -378,12 +386,14 @@ export async function findRequestQueueByPossibleId(client: MemoryStorage, entryNnewClient.modifiedAt = modifiedAt;newClient.pendingRequestCount = pendingRequestCount;newClient.handledRequestCount = handledRequestCount;+ // Drop any persisted forefront ids whose request file is missing or unparseable on disk (and was+ // therefore not added to `entries`). Keeping them would leave a dangling id in `forefrontRequestIds`+ // that `listPendingHead` would resolve to a missing request and dereference as `undefined`.// @ts-expect-error - Assigning to private property- newClient.forefrontRequestIds = forefrontRequestIds;+ newClient.forefrontRequestIds = forefrontRequestIds.filter((requestId) => entries.has(requestId));for (const requestId of entries) {const entry = new RequestQueueFileSystemEntry({- persistStorage: true,requestId,storeDirectory: requestQueueDir,});diff --git a/packages/fs-storage/src/consts.ts b/packages/fs-storage/src/consts.tsnew file mode 100644index 000000000..db8b29760--- /dev/null+++ b/packages/fs-storage/src/consts.ts@@ -0,0 +1,4 @@+/**+ * Length of id property of a Request instance in characters.+ */+export const REQUEST_ID_LENGTH = 15;diff --git a/packages/fs-storage/src/file-system-storage.ts b/packages/fs-storage/src/file-system-storage.tsnew file mode 100644index 000000000..ac427c9d3--- /dev/null+++ b/packages/fs-storage/src/file-system-storage.ts@@ -0,0 +1,434 @@+/* eslint-disable import/no-duplicates */+import { access, readdir, rm } from 'node:fs/promises';+import { resolve } from 'node:path';++import type * as storage from '@crawlee/types';+import type { CrawleeLogger } from '@crawlee/types';+import { s } from '@sapphire/shapeshift';+import { ensureDirSync, move, moveSync, pathExistsSync } from 'fs-extra/esm';++import { promiseMap, scheduleBackgroundTask } from './background-handler/index.js';+import {+ findOrCacheDatasetByPossibleId,+ findOrCacheKeyValueStoreByPossibleId,+ findRequestQueueByPossibleId,+} from './cache-helpers.js';+import { DatasetClient } from './resource-clients/dataset.js';+import { KeyValueStoreClient } from './resource-clients/key-value-store.js';+import { RequestQueueClient } from './resource-clients/request-queue.js';++export interface FileSystemStorageOptions {+ /**+ * Path to directory where the data will be saved.+ * @default process.env.CRAWLEE_STORAGE_DIR ?? './storage'+ */+ localDataDirectory?: string;++ /**+ * Whether to also write optional metadata files when storing to disk.+ * @default process.env.DEBUG?.includes('*') ?? process.env.DEBUG?.includes('crawlee:memory-storage') ?? false+ */+ writeMetadata?: boolean;++ /**+ * Optional logger for FileSystemStorageClient warnings.+ */+ logger?: CrawleeLogger;+}++export class FileSystemStorageClient implements storage.StorageClient {+ readonly localDataDirectory: string;+ readonly datasetsDirectory: string;+ readonly keyValueStoresDirectory: string;+ readonly requestQueuesDirectory: string;+ readonly writeMetadata: boolean;+ readonly logger?: CrawleeLogger;++ readonly keyValueStoreCache: KeyValueStoreClient[] = [];+ readonly datasetClientCache: DatasetClient[] = [];+ readonly requestQueueCache: RequestQueueClient[] = [];++ constructor(options: FileSystemStorageOptions = {}) {+ s.object({+ localDataDirectory: s.string().optional(),+ writeMetadata: s.boolean().optional(),+ }).parse(options);++ this.logger = options.logger;++ // v3.0.0 used `crawlee_storage` as the default, we changed this in v3.0.1 to just `storage`,+ // this function handles it without making BC breaks - it respects existing `crawlee_storage`+ // directories, and uses the `storage` only if it's not there.+ const defaultStorageDir = () => {+ if (pathExistsSync(resolve('./crawlee_storage'))) {+ return './crawlee_storage';+ }++ return './storage';+ };++ this.localDataDirectory = options.localDataDirectory ?? process.env.CRAWLEE_STORAGE_DIR ?? defaultStorageDir();+ this.datasetsDirectory = resolve(this.localDataDirectory, 'datasets');+ this.keyValueStoresDirectory = resolve(this.localDataDirectory, 'key_value_stores');+ this.requestQueuesDirectory = resolve(this.localDataDirectory, 'request_queues');+ this.writeMetadata =+ options.writeMetadata ??+ process.env.DEBUG?.includes('*') ??+ process.env.DEBUG?.includes('crawlee:memory-storage') ??+ false;+ }++ /**+ * Return a cache key that includes the resolved storage directory, so that+ * two `FileSystemStorageClient` instances pointing at different directories get separate+ * cache partitions. Mirrors crawlee-python's `FileSystemStorageClient` which+ * includes `configuration.storage_dir` in its cache key.+ */+ getStorageClientCacheKey(): string {+ return `FileSystemStorageClient:${resolve(this.localDataDirectory)}`;+ }++ private static resolveStorageKey(options: { id?: string; name?: string; alias?: string }): {+ isAlias: boolean;+ directoryKey: string | undefined;+ } {+ const isAlias = 'alias' in options && !!options.alias;+ const rawKey = isAlias ? options.alias : (options.name ?? options.id);+ // Normalize the internal __default__ alias to the user-facing 'default' name.+ const directoryKey = rawKey === '__default__' ? 'default' : rawKey;+ return { isAlias, directoryKey };+ }++ async createDatasetClient(options: storage.CreateDatasetClientOptions = {}): Promise<storage.DatasetClient> {+ const { isAlias, directoryKey } = FileSystemStorageClient.resolveStorageKey(options);++ if (directoryKey) {+ const found = await findOrCacheDatasetByPossibleId(this, directoryKey);+ if (found) {+ return found;+ }+ }++ const newStore = new DatasetClient({+ name: isAlias ? undefined : directoryKey,+ directoryName: directoryKey,+ baseStorageDirectory: this.datasetsDirectory,+ client: this,+ });+ this.datasetClientCache.push(newStore);++ // Schedule the worker to write to the disk+ const datasetInfo = newStore.toDatasetInfo();++ scheduleBackgroundTask(+ {+ action: 'update-metadata',+ entityType: 'datasets',+ entityDirectory: newStore.datasetDirectory,+ id: datasetInfo.name ?? datasetInfo.id,+ data: datasetInfo,+ writeMetadata: this.writeMetadata,+ },+ this.logger,+ );++ return newStore;+ }++ async createKeyValueStoreClient(+ options: storage.CreateKeyValueStoreClientOptions = {},+ ): Promise<storage.KeyValueStoreClient> {+ const { isAlias, directoryKey } = FileSystemStorageClient.resolveStorageKey(options);++ if (directoryKey) {+ const found = await findOrCacheKeyValueStoreByPossibleId(this, directoryKey);+ if (found) {+ return found;+ }+ }++ const newStore = new KeyValueStoreClient({+ name: isAlias ? undefined : directoryKey,+ directoryName: directoryKey,+ baseStorageDirectory: this.keyValueStoresDirectory,+ client: this,+ });+ this.keyValueStoreCache.push(newStore);++ // Schedule the worker to write to the disk+ const kvStoreInfo = newStore.toKeyValueStoreInfo();++ scheduleBackgroundTask(+ {+ action: 'update-metadata',+ entityType: 'keyValueStores',+ entityDirectory: newStore.keyValueStoreDirectory,+ id: kvStoreInfo.name ?? kvStoreInfo.id,+ data: kvStoreInfo,+ writeMetadata: this.writeMetadata,+ },+ this.logger,+ );++ return newStore;+ }++ async createRequestQueueClient(+ options: storage.CreateRequestQueueClientOptions = {},+ ): Promise<storage.RequestQueueClient> {+ const { isAlias, directoryKey } = FileSystemStorageClient.resolveStorageKey(options);++ if (directoryKey) {+ const found = await findRequestQueueByPossibleId(this, directoryKey);+ if (found) {+ return found;+ }+ }++ const newStore = new RequestQueueClient({+ name: isAlias ? undefined : directoryKey,+ directoryName: directoryKey,+ baseStorageDirectory: this.requestQueuesDirectory,+ client: this,+ });+ this.requestQueueCache.push(newStore);++ // Schedule the worker to write to the disk+ const queueInfo = newStore.toRequestQueueInfo();++ scheduleBackgroundTask(+ {+ action: 'update-metadata',+ entityType: 'requestQueues',+ entityDirectory: newStore.requestQueueDirectory,+ id: queueInfo.name ?? queueInfo.id,+ data: queueInfo,+ writeMetadata: this.writeMetadata,+ },+ this.logger,+ );++ return newStore;+ }++ async storageExists(id: string, type: 'Dataset' | 'KeyValueStore' | 'RequestQueue'): Promise<boolean> {+ let clients: { id: string }[];+ let baseDir: string;++ switch (type) {+ case 'Dataset':+ clients = this.datasetClientCache;+ baseDir = this.datasetsDirectory;+ break;+ case 'KeyValueStore':+ clients = this.keyValueStoreCache;+ baseDir = this.keyValueStoresDirectory;+ break;+ case 'RequestQueue':+ clients = this.requestQueueCache;+ baseDir = this.requestQueuesDirectory;+ break;+ default:+ return false;+ }++ // Check in-memory cache by actual storage ID+ if (clients.some((store) => store.id === id)) {+ return true;+ }++ // Check if a directory with that ID exists on disk.+ // Only consider directories whose name matches the queried ID — this avoids+ // false positives for alias-created directories (e.g. a directory named 'asdf'+ // created via `{ alias: 'asdf' }` should not make `storageExists('asdf')` return true,+ // since the actual storage ID is a UUID, not the alias string).+ try {+ await access(resolve(baseDir, id));++ // If the directory exists but a cached client already owns this directory+ // under a different ID, this is not a match.+ const cachedClients = clients as { id: string; directoryName?: string }[];+ if (cachedClients.some((store) => store.directoryName === id && store.id !== id)) {+ return false;+ }++ return true;+ } catch {+ return false;+ }+ }++ async setStatusMessage(message: string, options: storage.SetStatusMessageOptions = {}): Promise<void> {+ s.string().parse(message);+ s.object({+ isStatusMessageTerminal: s.boolean().optional(),+ }).parse(options);++ return Promise.resolve();+ }++ /**+ * Cleans up the default storage directories before the run starts:+ * - local directory containing the default dataset;+ * - all records from the default key-value store in the local directory, except for the "INPUT" key;+ * - local directory containing the default request queue.+ */+ async purge(): Promise<void> {+ // Key-value stores+ const keyValueStores = await readdir(this.keyValueStoresDirectory).catch(() => []);+ const keyValueStorePromises: Promise<void>[] = [];++ for (const keyValueStoreFolder of keyValueStores) {+ if (keyValueStoreFolder.startsWith('__CRAWLEE_TEMPORARY') || keyValueStoreFolder.startsWith('__OLD')) {+ keyValueStorePromises.push(+ (await this.batchRemoveFiles(resolve(this.keyValueStoresDirectory, keyValueStoreFolder)))(),+ );+ } else if (keyValueStoreFolder === 'default' || keyValueStoreFolder === '__default__') {+ keyValueStorePromises.push(+ this.handleDefaultKeyValueStore(resolve(this.keyValueStoresDirectory, keyValueStoreFolder))(),+ );+ }+ }++ void Promise.allSettled(keyValueStorePromises);++ // Datasets+ const datasets = await readdir(this.datasetsDirectory).catch(() => []);+ const datasetPromises: Promise<void>[] = [];++ for (const datasetFolder of datasets) {+ if (+ datasetFolder === 'default' ||+ datasetFolder === '__default__' ||+ datasetFolder.startsWith('__CRAWLEE_TEMPORARY')+ ) {+ datasetPromises.push((await this.batchRemoveFiles(resolve(this.datasetsDirectory, datasetFolder)))());+ }+ }++ void Promise.allSettled(datasetPromises);++ // Request queues+ const requestQueues = await readdir(this.requestQueuesDirectory).catch(() => []);+ const requestQueuePromises: Promise<void>[] = [];++ for (const requestQueueFolder of requestQueues) {+ if (+ requestQueueFolder === 'default' ||+ requestQueueFolder === '__default__' ||+ requestQueueFolder.startsWith('__CRAWLEE_TEMPORARY')+ ) {+ requestQueuePromises.push(+ (await this.batchRemoveFiles(resolve(this.requestQueuesDirectory, requestQueueFolder)))(),+ );+ }+ }++ void Promise.allSettled(requestQueuePromises);+ }++ /**+ * This method should be called at the end of the process, to ensure all data is saved.+ */+ async teardown(): Promise<void> {+ // Release any request locks this process still holds so that requests fetched but not yet+ // handled are not stuck (until their lock expires) for the next consumer of the same on-disk+ // queue. Other storage backends don't need this: the Apify platform releases a run's locks+ // automatically, and the file-system storage doesn't lock at all.+ await Promise.all(this.requestQueueCache.map(async (queue) => queue.releaseOwnLocks()));++ const promises = [...promiseMap.values()].map(async ({ promise }) => promise);++ await Promise.all(promises);+ }++ private handleDefaultKeyValueStore(folder: string): () => Promise<void> {+ const storagePathExists = pathExistsSync(folder);+ const temporaryPath = resolve(folder, '../__CRAWLEE_MIGRATING_KEY_VALUE_STORE__');++ // For optimization, we want to only attempt to copy a few files from the default key-value store+ const possibleInputKeys = ['INPUT', 'INPUT.json', 'INPUT.bin', 'INPUT.txt'];++ if (storagePathExists) {+ // Create temporary folder to save important files in+ ensureDirSync(temporaryPath);++ // Go through each file and save the ones that are important+ for (const entity of possibleInputKeys) {+ const originalFilePath = resolve(folder, entity);+ const tempFilePath = resolve(temporaryPath, entity);++ try {+ moveSync(originalFilePath, tempFilePath);+ } catch {+ // Ignore+ }+ }++ // Remove the original folder and all its content+ let counter = 0;+ let tempPathForOldFolder = resolve(folder, `../__OLD_DEFAULT_${counter}__`);+ let done = false;++ while (!done) {+ try {+ moveSync(folder, tempPathForOldFolder);+ done = true;+ } catch {+ tempPathForOldFolder = resolve(folder, `../__OLD_DEFAULT_${++counter}__`);+ }+ }++ // Replace the temporary folder with the original folder+ moveSync(temporaryPath, folder);++ // Remove the old folder+ return async () => (await this.batchRemoveFiles(tempPathForOldFolder))();+ }++ return async () => Promise.resolve();+ }++ private async batchRemoveFiles(folder: string, counter = 0): Promise<() => Promise<void>> {+ const folderExists = pathExistsSync(folder);++ if (folderExists) {+ const temporaryFolder = resolve(folder, `../__CRAWLEE_TEMPORARY_${counter}__`);++ try {+ // Rename the old folder to the new one to allow background deletions+ await move(folder, temporaryFolder);+ } catch {+ // Folder exists already, try again with an incremented counter+ return this.batchRemoveFiles(folder, ++counter);+ }++ return async () => {+ // Read all files in the folder+ const entries = await readdir(temporaryFolder);++ let processed = 0;+ let promises: Promise<void>[] = [];++ for (const entry of entries) {+ processed++;+ promises.push(rm(resolve(temporaryFolder, entry), { force: true }));++ // Every 2000 files, delete them+ if (processed % 2000 === 0) {+ await Promise.allSettled(promises);+ promises = [];+ }+ }++ // Ensure last promises are handled+ await Promise.allSettled(promises);++ // Delete the folder itself+ await rm(temporaryFolder, { force: true, recursive: true });+ };+ }++ return async () => Promise.resolve();+ }+}diff --git a/packages/memory-storage/src/fs/common.ts b/packages/fs-storage/src/fs/common.tssimilarity index 100%rename from packages/memory-storage/src/fs/common.tsrename to packages/fs-storage/src/fs/common.tsdiff --git a/packages/memory-storage/src/fs/dataset/fs.ts b/packages/fs-storage/src/fs/dataset/fs.tssimilarity index 100%rename from packages/memory-storage/src/fs/dataset/fs.tsrename to packages/fs-storage/src/fs/dataset/fs.tsdiff --git a/packages/memory-storage/src/fs/dataset/index.ts b/packages/fs-storage/src/fs/dataset/index.tssimilarity index 68%rename from packages/memory-storage/src/fs/dataset/index.tsrename to packages/fs-storage/src/fs/dataset/index.tsindex 2d61c910d..bd1f71cee 100644--- a/packages/memory-storage/src/fs/dataset/index.ts+++ b/packages/fs-storage/src/fs/dataset/index.ts@@ -2,20 +2,14 @@ import type { Dictionary } from '@crawlee/types';import type { StorageImplementation } from '../common.js';import { DatasetFileSystemEntry } from './fs.js';-import { DatasetMemoryEntry } from './memory.js';export function createDatasetStorageImplementation<Data extends Dictionary>(options: CreateStorageImplementationOptions,): StorageImplementation<Data> {- if (options.persistStorage) {- return new DatasetFileSystemEntry<Data>(options);- }-- return new DatasetMemoryEntry<Data>();+ return new DatasetFileSystemEntry<Data>(options);}export interface CreateStorageImplementationOptions {- persistStorage: boolean;storeDirectory: string;/** The actual id of the file to save */entityId: string;diff --git a/packages/memory-storage/src/fs/key-value-store/fs.ts b/packages/fs-storage/src/fs/key-value-store/fs.tssimilarity index 97%rename from packages/memory-storage/src/fs/key-value-store/fs.tsrename to packages/fs-storage/src/fs/key-value-store/fs.tsindex 6af3e567c..e0d662801 100644--- a/packages/memory-storage/src/fs/key-value-store/fs.ts+++ b/packages/fs-storage/src/fs/key-value-store/fs.ts@@ -36,7 +36,7 @@ export class KeyValueFileSystemEntry implements StorageImplementation<InternalKefile = await readFile(this.filePath);} catch {try {- const noExtFilePath = resolve(this.storeDirectory, this.rawRecord.key);+ const noExtFilePath = resolveWithinDirectory(this.storeDirectory, this.rawRecord.key);// Try without extensionfile = await readFile(noExtFilePath);this.logger?.warning(diff --git a/packages/memory-storage/src/fs/key-value-store/index.ts b/packages/fs-storage/src/fs/key-value-store/index.tssimilarity index 71%rename from packages/memory-storage/src/fs/key-value-store/index.tsrename to packages/fs-storage/src/fs/key-value-store/index.tsindex 6947b2b10..789d15d2f 100644--- a/packages/memory-storage/src/fs/key-value-store/index.ts+++ b/packages/fs-storage/src/fs/key-value-store/index.ts@@ -3,20 +3,14 @@ import type { CrawleeLogger } from '@crawlee/types';import type { InternalKeyRecord } from '../../resource-clients/key-value-store.js';import type { StorageImplementation } from '../common.js';import { KeyValueFileSystemEntry } from './fs.js';-import { KeyValueMemoryEntry } from './memory.js';export function createKeyValueStorageImplementation(options: CreateStorageImplementationOptions,): StorageImplementation<InternalKeyRecord> {- if (options.persistStorage) {- return new KeyValueFileSystemEntry(options);- }-- return new KeyValueMemoryEntry();+ return new KeyValueFileSystemEntry(options);}export interface CreateStorageImplementationOptions {- persistStorage: boolean;storeDirectory: string;writeMetadata: boolean;logger?: CrawleeLogger;diff --git a/packages/memory-storage/src/fs/request-queue/fs.ts b/packages/fs-storage/src/fs/request-queue/fs.tssimilarity index 100%rename from packages/memory-storage/src/fs/request-queue/fs.tsrename to packages/fs-storage/src/fs/request-queue/fs.tsdiff --git a/packages/memory-storage/src/fs/request-queue/index.ts b/packages/fs-storage/src/fs/request-queue/index.tssimilarity index 54%rename from packages/memory-storage/src/fs/request-queue/index.tsrename to packages/fs-storage/src/fs/request-queue/index.tsindex cb903bb6e..995c399ad 100644--- a/packages/memory-storage/src/fs/request-queue/index.ts+++ b/packages/fs-storage/src/fs/request-queue/index.ts@@ -1,16 +1,10 @@import { RequestQueueFileSystemEntry } from './fs.js';-import { RequestQueueMemoryEntry } from './memory.js';export function createRequestQueueStorageImplementation(options: CreateStorageImplementationOptions) {- if (options.persistStorage) {- return new RequestQueueFileSystemEntry(options);- }-- return new RequestQueueMemoryEntry();+ return new RequestQueueFileSystemEntry(options);}export interface CreateStorageImplementationOptions {- persistStorage: boolean;storeDirectory: string;requestId: string;}diff --git a/packages/fs-storage/src/index.ts b/packages/fs-storage/src/index.tsnew file mode 100644index 000000000..9219ced98--- /dev/null+++ b/packages/fs-storage/src/index.ts@@ -0,0 +1 @@+export * from './file-system-storage.js';diff --git a/packages/fs-storage/src/resource-clients/common/base-client.ts b/packages/fs-storage/src/resource-clients/common/base-client.tsnew file mode 100644index 000000000..72a994c2d--- /dev/null+++ b/packages/fs-storage/src/resource-clients/common/base-client.ts@@ -0,0 +1,7 @@+export class BaseClient {+ id: string;++ constructor(id: string) {+ this.id = id;+ }+}diff --git a/packages/fs-storage/src/resource-clients/dataset.ts b/packages/fs-storage/src/resource-clients/dataset.tsnew file mode 100644index 000000000..20ec7baa2--- /dev/null+++ b/packages/fs-storage/src/resource-clients/dataset.ts@@ -0,0 +1,202 @@+/* eslint-disable import/no-duplicates */+import { randomUUID } from 'node:crypto';+import { rm } from 'node:fs/promises';+import { resolve } from 'node:path';++import type * as storage from '@crawlee/types';+import type { Dictionary } from '@crawlee/types';+import { s } from '@sapphire/shapeshift';++import { scheduleBackgroundTask } from '../background-handler/index.js';+import type { StorageImplementation } from '../fs/common.js';+import { createDatasetStorageImplementation } from '../fs/dataset/index.js';+import type { FileSystemStorageClient } from '../index.js';+import { resolveWithinDirectory } from '../utils.js';+import { BaseClient } from './common/base-client.js';++/**+ * This is what API returns in the x-apify-pagination-limit+ * header when no limit query parameter is used.+ */+const LIST_ITEMS_LIMIT = 999_999_999_999;++/**+ * Number of characters of the dataset item file names.+ * E.g.: 000000019.json - 9 digits+ */+const LOCAL_ENTRY_NAME_DIGITS = 9;++export interface DatasetClientOptions {+ id?: string;+ name?: string;+ /**+ * The directory name to use on disk. When provided, takes precedence over `name` and `id`+ * for the directory path. This allows alias-opened storages to have a directory name+ * that differs from their metadata `name` (which is `undefined` for unnamed storages).+ */+ directoryName?: string;+ baseStorageDirectory: string;+ client: FileSystemStorageClient;+}++export class DatasetClient<Data extends Dictionary = Dictionary>+ extends BaseClient+ implements storage.DatasetClient<Data>+{+ name?: string;+ /**+ * The key used for directory naming and cache lookup. For named storages, this equals+ * the name. For alias (unnamed) storages, this is the alias string. Falls back to id.+ */+ directoryName: string;+ createdAt = new Date();+ accessedAt = new Date();+ modifiedAt = new Date();+ itemCount = 0;+ datasetDirectory: string;++ private readonly datasetEntries = new Map<string, StorageImplementation<Data>>();+ private readonly client: FileSystemStorageClient;++ constructor(options: DatasetClientOptions) {+ super(options.id ?? randomUUID());+ this.name = options.name;+ this.directoryName = options.directoryName ?? this.name ?? this.id;+ this.datasetDirectory = resolveWithinDirectory(options.baseStorageDirectory, this.directoryName);+ this.client = options.client;+ }++ async getMetadata(): Promise<storage.DatasetInfo> {+ this.updateTimestamps(false);+ return this.toDatasetInfo();+ }++ async drop(): Promise<void> {+ const storeIndex = this.client.datasetClientCache.findIndex((store) => store.id === this.id);++ if (storeIndex !== -1) {+ const [oldClient] = this.client.datasetClientCache.splice(storeIndex, 1);+ oldClient.itemCount = 0;+ oldClient.datasetEntries.clear();++ await rm(oldClient.datasetDirectory, { recursive: true, force: true });+ }+ }++ async purge(): Promise<void> {+ this.itemCount = 0;+ this.datasetEntries.clear();++ // Remove item files from disk but keep the directory+ const { readdir } = await import('node:fs/promises');+ const entries = await readdir(this.datasetDirectory).catch(() => []);+ for (const entry of entries) {+ if (entry !== '__metadata__.json') {+ await rm(resolve(this.datasetDirectory, entry), { force: true });+ }+ }++ this.updateTimestamps(true);+ }++ getData(options: storage.DatasetClientListOptions = {}): Promise<storage.PaginatedList<Data>> {+ const { desc, limit, offset } = s+ .object({+ desc: s.boolean().optional(),+ limit: s.number().int().optional(),+ offset: s.number().int().optional(),+ })+ .parse(options);++ return this.getDataPage({+ desc,+ offset: offset ?? 0,+ limit: Math.min(limit ?? LIST_ITEMS_LIMIT, LIST_ITEMS_LIMIT),+ });+ }++ private async getDataPage(options: storage.DatasetClientListOptions = {}): Promise<storage.PaginatedList<Data>> {+ const { limit = LIST_ITEMS_LIMIT, offset = 0, desc } = options;++ const [start, end] = this.getStartAndEndIndexes(+ desc ? Math.max(this.itemCount - offset - limit, 0) : offset,+ limit,+ );++ const items: Data[] = [];++ for (let idx = start; idx < end; idx++) {+ const entryNumber = this.generateLocalEntryName(idx);+ items.push(await this.datasetEntries.get(entryNumber)!.get());+ }++ this.updateTimestamps(false);++ return {+ count: items.length,+ desc: desc ?? false,+ items: desc ? items.reverse() : items,+ limit,+ offset,+ total: this.itemCount,+ };+ }++ async pushData(items: Data[]): Promise<void> {+ for (const entry of items) {+ const idx = this.generateLocalEntryName(++this.itemCount);+ const storageEntry = createDatasetStorageImplementation({+ entityId: idx,+ storeDirectory: this.datasetDirectory,+ });++ await storageEntry.update(entry);++ this.datasetEntries.set(idx, storageEntry);+ }++ this.updateTimestamps(true);+ }++ toDatasetInfo(): storage.DatasetInfo {+ return {+ id: this.id,+ accessedAt: this.accessedAt,+ createdAt: this.createdAt,+ itemCount: this.itemCount,+ modifiedAt: this.modifiedAt,+ name: this.name,+ };+ }++ private generateLocalEntryName(idx: number): string {+ return idx.toString().padStart(LOCAL_ENTRY_NAME_DIGITS, '0');+ }++ private getStartAndEndIndexes(offset: number, limit = this.itemCount) {+ const start = offset + 1;+ const end = Math.min(offset + limit, this.itemCount) + 1;+ return [start, end] as const;+ }++ private updateTimestamps(hasBeenModified: boolean) {+ this.accessedAt = new Date();++ if (hasBeenModified) {+ this.modifiedAt = new Date();+ }++ const data = this.toDatasetInfo();+ scheduleBackgroundTask(+ {+ action: 'update-metadata',+ data,+ entityType: 'datasets',+ entityDirectory: this.datasetDirectory,+ id: this.name ?? this.id,+ writeMetadata: this.client.writeMetadata,+ },+ this.client.logger,+ );+ }+}diff --git a/packages/fs-storage/src/resource-clients/key-value-store.ts b/packages/fs-storage/src/resource-clients/key-value-store.tsnew file mode 100644index 000000000..be02b7cba--- /dev/null+++ b/packages/fs-storage/src/resource-clients/key-value-store.ts@@ -0,0 +1,300 @@+import { randomUUID } from 'node:crypto';+import { rm } from 'node:fs/promises';+import { pathToFileURL } from 'node:url';++import type * as storage from '@crawlee/types';+import { s } from '@sapphire/shapeshift';++import { scheduleBackgroundTask } from '../background-handler/index.js';+import { maybeParseBody } from '../body-parser.js';+import type { StorageImplementation } from '../fs/common.js';+import { createKeyValueStorageImplementation } from '../fs/key-value-store/index.js';+import type { FileSystemStorageClient } from '../index.js';+import { isBuffer, isStream, resolveWithinDirectory } from '../utils.js';+import { BaseClient } from './common/base-client.js';+import mime from 'mime-types';++const DEFAULT_LOCAL_FILE_EXTENSION = 'bin';++export interface KeyValueStoreClientOptions {+ name?: string;+ id?: string;+ /**+ * The directory name to use on disk. When provided, takes precedence over `name` and `id`+ * for the directory path. This allows alias-opened storages to have a directory name+ * that differs from their metadata `name` (which is `undefined` for unnamed storages).+ */+ directoryName?: string;+ baseStorageDirectory: string;+ client: FileSystemStorageClient;+}++export interface InternalKeyRecord {+ key: string;+ value: Buffer | string;+ contentType?: string;+ extension: string;+ filePath?: string;+}++export class KeyValueStoreClient extends BaseClient implements storage.KeyValueStoreClient {+ name?: string;+ /**+ * The key used for directory naming and cache lookup. For named storages, this equals+ * the name. For alias (unnamed) storages, this is the alias string. Falls back to id.+ */+ directoryName: string;+ createdAt = new Date();+ accessedAt = new Date();+ modifiedAt = new Date();+ keyValueStoreDirectory: string;++ private readonly keyValueEntries = new Map<string, StorageImplementation<InternalKeyRecord>>();+ private readonly client: FileSystemStorageClient;++ constructor(options: KeyValueStoreClientOptions) {+ super(options.id ?? randomUUID());+ this.name = options.name;+ this.directoryName = options.directoryName ?? this.name ?? this.id;+ this.keyValueStoreDirectory = resolveWithinDirectory(options.baseStorageDirectory, this.directoryName);+ this.client = options.client;+ }++ async getMetadata(): Promise<storage.KeyValueStoreInfo> {+ this.updateTimestamps(false);+ return this.toKeyValueStoreInfo();+ }++ async drop(): Promise<void> {+ const storeIndex = this.client.keyValueStoreCache.findIndex((store) => store.id === this.id);++ if (storeIndex !== -1) {+ const [oldClient] = this.client.keyValueStoreCache.splice(storeIndex, 1);+ oldClient.keyValueEntries.clear();++ await rm(oldClient.keyValueStoreDirectory, { recursive: true, force: true });+ }+ }++ async purge(): Promise<void> {+ // Delete all entries+ const entriesToDelete = [...this.keyValueEntries.entries()];+ for (const [key, entry] of entriesToDelete) {+ this.keyValueEntries.delete(key);+ await entry.delete();+ }++ this.updateTimestamps(true);+ }++ async listKeys(options: storage.KeyValueStoreListKeysOptions = {}): Promise<storage.KeyValueStoreItemData[]> {+ const { prefix, exclusiveStartKey, limit } = s+ .object({+ prefix: s.string().optional(),+ exclusiveStartKey: s.string().optional(),+ limit: s.number().int().greaterThan(0).optional(),+ })+ .parse(options);++ const items: storage.KeyValueStoreItemData[] = [];++ for (const storageEntry of this.keyValueEntries.values()) {+ const record = await storageEntry.get();++ const size = Buffer.byteLength(record.value);+ items.push({+ key: record.key,+ size,+ });+ }++ // Lexically sort to emulate API.+ items.sort((a, b) => a.key.localeCompare(b.key));++ let filteredItems = items.filter((item) => !prefix || item.key.startsWith(prefix));++ if (exclusiveStartKey) {+ const keyPos = filteredItems.findIndex((item) => item.key === exclusiveStartKey);+ if (keyPos === -1) {+ throw new Error(+ `exclusiveStartKey "${exclusiveStartKey}" was not found in the key-value store. ` ++ `This is likely a bug — the key may have been deleted between paginated listKeys calls.`,+ );+ }+ filteredItems = filteredItems.slice(keyPos + 1);+ }++ if (limit !== undefined) {+ filteredItems = filteredItems.slice(0, limit);+ }++ this.updateTimestamps(false);++ return filteredItems;+ }++ /**+ * Generates a public `file://` URL for accessing a specific record in the key-value store.+ *+ * Returns `undefined` if the record does not exist or has no associated file path (i.e., it is not stored as a file).+ * @param key The key of the record to generate the public URL for.+ */+ async getPublicUrl(key: string): Promise<string | undefined> {+ s.string().parse(key);++ const storageEntry = await this.keyValueEntries.get(key)?.get();++ return storageEntry?.filePath ? pathToFileURL(storageEntry.filePath).href : undefined;+ }++ /**+ * Tests whether a record with the given key exists in the key-value store without retrieving its value.+ *+ * @param key The queried record key.+ * @returns `true` if the record exists, `false` if it does not.+ */+ async recordExists(key: string): Promise<boolean> {+ s.string().parse(key);++ return this.keyValueEntries.has(key);+ }++ async getValue(key: string): Promise<storage.KeyValueStoreRecord | undefined> {+ s.string().parse(key);++ const storageEntry = this.keyValueEntries.get(key);++ if (!storageEntry) {+ return undefined;+ }++ const entry = await storageEntry.get();++ const record: storage.KeyValueStoreRecord = {+ key: entry.key,+ value: entry.value,+ contentType: entry.contentType ?? (mime.contentType(entry.extension) as string),+ };++ // Auto-parse the body (JSON → object, text → string, etc.)+ record.value = maybeParseBody(record.value, record.contentType!);++ this.updateTimestamps(false);++ return record;+ }++ async setValue(record: storage.KeyValueStoreRecord): Promise<void> {+ s.object({+ key: s.string().lengthGreaterThan(0),+ value: s.union([+ s.null(),+ s.string(),+ s.number(),+ s.instance(Buffer),+ s.instance(ArrayBuffer),+ s.typedArray(),+ // disabling validation will make shapeshift only check the object given is an actual object, not null, nor array+ s.object({}).setValidationEnabled(false),+ ]),+ contentType: s.string().lengthGreaterThan(0).optional(),+ }).parse(record);++ const { key } = record;+ let { value, contentType } = record;++ const valueIsStream = isStream(value);++ const isValueStreamOrBuffer = valueIsStream || isBuffer(value);+ // To allow saving Objects to JSON without providing content type+ if (!contentType) {+ if (isValueStreamOrBuffer) contentType = 'application/octet-stream';+ else if (typeof value === 'string') contentType = 'text/plain; charset=utf-8';+ else contentType = 'application/json; charset=utf-8';+ }++ const extension = mime.extension(contentType) || DEFAULT_LOCAL_FILE_EXTENSION;++ const isContentTypeJson = extension === 'json';++ if (isContentTypeJson && !isValueStreamOrBuffer && typeof value !== 'string') {+ try {+ value = JSON.stringify(value, null, 2);+ } catch (err: any) {+ const msg = `The record value cannot be stringified to JSON. Please provide other content type.\nCause: ${err.message}`;+ throw new Error(msg);+ }+ }++ if (valueIsStream) {+ const chunks = [];+ for await (const chunk of value) {+ chunks.push(chunk);+ }+ value = Buffer.concat(chunks);+ }++ const _record = {+ extension,+ key,+ value,+ contentType,+ } satisfies InternalKeyRecord;++ const entry = createKeyValueStorageImplementation({+ storeDirectory: this.keyValueStoreDirectory,+ writeMetadata: this.client.writeMetadata,+ logger: this.client.logger,+ });++ await entry.update(_record);++ this.keyValueEntries.set(key, entry);++ this.updateTimestamps(true);+ }++ async deleteValue(key: string): Promise<void> {+ s.string().parse(key);++ const entry = this.keyValueEntries.get(key);++ if (entry) {+ this.keyValueEntries.delete(key);+ this.updateTimestamps(true);+ await entry.delete();+ }+ }++ toKeyValueStoreInfo(): storage.KeyValueStoreInfo {+ return {+ id: this.id,+ name: this.name,+ accessedAt: this.accessedAt,+ createdAt: this.createdAt,+ modifiedAt: this.modifiedAt,+ userId: '1',+ };+ }++ private updateTimestamps(hasBeenModified: boolean) {+ this.accessedAt = new Date();++ if (hasBeenModified) {+ this.modifiedAt = new Date();+ }++ const data = this.toKeyValueStoreInfo();+ scheduleBackgroundTask(+ {+ action: 'update-metadata',+ data,+ entityType: 'keyValueStores',+ entityDirectory: this.keyValueStoreDirectory,+ id: this.name ?? this.id,+ writeMetadata: this.client.writeMetadata,+ },+ this.client.logger,+ );+ }+}diff --git a/packages/fs-storage/src/resource-clients/request-queue.ts b/packages/fs-storage/src/resource-clients/request-queue.tsnew file mode 100644index 000000000..de63231d2--- /dev/null+++ b/packages/fs-storage/src/resource-clients/request-queue.ts@@ -0,0 +1,682 @@+import { randomUUID } from 'node:crypto';+import { rm } from 'node:fs/promises';+import { resolve } from 'node:path';++import type * as storage from '@crawlee/types';+import { AsyncQueue } from '@sapphire/async-queue';+import { s } from '@sapphire/shapeshift';+import type { RequestQueueFileSystemEntry } from '../fs/request-queue/fs.js';++import { scheduleBackgroundTask } from '../background-handler/index.js';+import { createRequestQueueStorageImplementation } from '../fs/request-queue/index.js';+import type { FileSystemStorageClient } from '../index.js';+import { purgeNullsFromObject, resolveWithinDirectory, uniqueKeyToRequestId } from '../utils.js';+import { BaseClient } from './common/base-client.js';++const requestShape = s+ .object({+ id: s.string(),+ url: s.string().url({ allowedProtocols: ['http:', 'https:'] }),+ uniqueKey: s.string(),+ method: s.string().optional(),+ retryCount: s.number().int().optional(),+ handledAt: s.union([s.string(), s.date().valid()]).optional(),+ })+ .passthrough();++const requestShapeWithoutId = requestShape.omit(['id']);++const batchRequestShapeWithoutId = requestShapeWithoutId.array();++const requestOptionsShape = s.object({+ forefront: s.boolean().optional(),+});++export interface RequestQueueClientOptions {+ name?: string;+ id?: string;+ /**+ * The directory name to use on disk. When provided, takes precedence over `name` and `id`+ * for the directory path. This allows alias-opened storages to have a directory name+ * that differs from their metadata `name` (which is `undefined` for unnamed storages).+ */+ directoryName?: string;+ baseStorageDirectory: string;+ client: FileSystemStorageClient;+}++export interface InternalRequest {+ id: string;+ orderNo: number | null;+ url: string;+ uniqueKey: string;+ method: Exclude<storage.RequestOptions['method'], undefined>;+ retryCount: number;+ json: string;+}++/**+ * Default time (in seconds) for which a request fetched via {@link RequestQueueClient.fetchNextRequest}+ * stays locked (in progress) before it becomes available again. Aligns with the historical request queue+ * locking default. A consumer (e.g. a crawler) can raise this per queue via+ * {@link RequestQueueClient.setExpectedRequestProcessingTimeSecs}.+ */+const DEFAULT_REQUEST_LOCK_SECS = 3 * 60;++/**+ * A request is "locked" (in progress) when its `orderNo` is pushed beyond the current time. The sign of+ * `orderNo` is preserved so the original forefront / normal ordering is restored once the lock expires+ * or the request is reclaimed.+ */+function isRequestLocked(orderNo: number | null, now: number): boolean {+ return orderNo !== null && Math.abs(orderNo) > now;+}++export class RequestQueueClient extends BaseClient implements storage.RequestQueueClient {+ name?: string;+ /**+ * The key used for directory naming and cache lookup. For named storages, this equals+ * the name. For alias (unnamed) storages, this is the alias string. Falls back to id.+ */+ directoryName: string;+ createdAt = new Date();+ accessedAt = new Date();+ modifiedAt = new Date();+ handledRequestCount = 0;+ pendingRequestCount = 0;+ requestQueueDirectory: string;+ /**+ * Serializes every operation that reads-then-writes this client's shared queue state — the+ * `requests` map, the `forefrontRequestIds` array, the `inProgressRequestIds` set and the request+ * counts. Those mutations span `await` points (disk/storage I/O), so without this lock a concurrent+ * operation could interleave and corrupt them (e.g. a head scan pruning `forefrontRequestIds` while+ * `addBatchOfRequests` pushes to it). Held by every mutating method as well as by `isEmpty`/+ * `isFinished`, whose head scan also prunes `forefrontRequestIds`.+ */+ private readonly queueStateMutex = new AsyncQueue();+ private forefrontRequestIds: string[] = [];++ /**+ * IDs of requests this client has locked (fetched but not yet handled or reclaimed). Used by+ * {@link releaseOwnLocks} to free our own in-progress requests on process termination, so that+ * a crashed/migrated consumer does not block its requests for the full lock duration.+ */+ private readonly inProgressRequestIds = new Set<string>();++ private readonly requests = new Map<string, RequestQueueFileSystemEntry>();+ private readonly client: FileSystemStorageClient;++ /**+ * How long (in seconds) a request fetched from this client stays locked (in progress). Defaults to+ * {@link DEFAULT_REQUEST_LOCK_SECS} and is overridable via {@link setExpectedRequestProcessingTimeSecs}.+ */+ private lockSecs = DEFAULT_REQUEST_LOCK_SECS;++ constructor(options: RequestQueueClientOptions) {+ super(options.id ?? randomUUID());+ this.name = options.name;+ this.directoryName = options.directoryName ?? this.name ?? this.id;+ this.requestQueueDirectory = resolveWithinDirectory(options.baseStorageDirectory, this.directoryName);+ this.client = options.client;+ }++ /**+ * Applies how long {@link fetchNextRequest} locks a request before it becomes available again. The+ * caller (the `RequestQueue` frontend) owns the policy of what this value should be — this method+ * just applies it.+ */+ setExpectedRequestProcessingTimeSecs(secs: number): void {+ this.lockSecs = secs;+ }++ async getMetadata(): Promise<storage.RequestQueueInfo> {+ this.updateTimestamps(false);+ return this.toRequestQueueInfo();+ }++ async drop(): Promise<void> {+ // Serialize against other mutators (and the head scans in `isEmpty`/`isFinished`) so a concurrent+ // operation cannot observe half-cleared state — e.g. a forefront id whose request has already been+ // removed, which `listPendingHead` would then dereference as `undefined`.+ await this.queueStateMutex.wait();++ try {+ const storeIndex = this.client.requestQueueCache.findIndex((queue) => queue.id === this.id);++ if (storeIndex !== -1) {+ const [oldClient] = this.client.requestQueueCache.splice(storeIndex, 1);+ oldClient.pendingRequestCount = 0;+ // Clear all in-memory state, consistent with `purge`. Clearing `requests` alone would+ // leave dangling ids in `forefrontRequestIds`/`inProgressRequestIds`, which a later head+ // scan would resolve to a missing request and dereference.+ oldClient.requests.clear();+ oldClient.forefrontRequestIds = [];+ oldClient.inProgressRequestIds.clear();++ await rm(oldClient.requestQueueDirectory, { recursive: true, force: true });+ }+ } finally {+ this.queueStateMutex.shift();+ }+ }++ async purge(): Promise<void> {+ // Serialize against other mutators (and the head scans in `isEmpty`/`isFinished`) so a concurrent+ // operation cannot observe or repopulate half-cleared state across the `await` below.+ await this.queueStateMutex.wait();++ try {+ // Clear all in-memory state+ this.requests.clear();+ this.forefrontRequestIds = [];+ this.inProgressRequestIds.clear();+ this.handledRequestCount = 0;+ this.pendingRequestCount = 0;++ // Reset the lock duration back to the default so a value raised via+ // `setExpectedRequestProcessingTimeSecs` in an earlier run does not leak into a later one+ this.lockSecs = DEFAULT_REQUEST_LOCK_SECS;++ // Remove request files from disk but keep the directory+ const { readdir } = await import('node:fs/promises');+ const entries = await readdir(this.requestQueueDirectory).catch(() => []);+ for (const entry of entries) {+ if (entry !== '__metadata__.json') {+ await rm(resolve(this.requestQueueDirectory, entry), { force: true });+ }+ }++ this.updateTimestamps(true);+ } finally {+ this.queueStateMutex.shift();+ }+ }++ private *requestKeyIterator(): IterableIterator<string> {+ for (let i = this.forefrontRequestIds.length - 1; i >= 0; i--) {+ yield this.forefrontRequestIds[i];+ }++ for (const key of this.requests.keys()) {+ yield key;+ }+ }++ /**+ * Scans the queue and returns the pending head — requests that are neither handled nor currently+ * locked (in progress) — ordered by `orderNo`, deduplicated.+ *+ * When `detectLockedRequests` is set, the result also carries a `hasLockedRequests` flag telling+ * whether any unhandled-but-locked request was skipped along the way. This mirrors the Apify+ * platform shared client's `queueHasLockedRequests` signal: it lets {@link isFinished} distinguish+ * "no work left at all" from "work remains, but it is currently locked by some consumer (possibly+ * another process)". Without it, a consumer would consider the queue finished and let the crawler+ * shut down while another consumer still holds the last requests.+ *+ * Computing the flag is expensive: because a lock may sit anywhere in the queue, it forces a scan+ * of every pending entry even when only `limit` items are wanted. Callers that only need the head+ * (e.g. {@link fetchNextRequest}, {@link isEmpty}) leave it off so the scan can stop as soon as the+ * page is filled, keeping those calls O(head) instead of O(N).+ *+ * Lock state lives in the persisted `orderNo` (see {@link isRequestLocked}), so that processes+ * sharing the same on-disk queue observe each other's locks. We therefore re-read entries from+ * storage to obtain fresh lock state, except for entries we can cheaply rule out as permanently+ * handled via their cached `orderNo === null`.+ */+ private async listPendingHead(+ limit: number,+ detectLockedRequests = false,+ ): Promise<{ items: InternalRequest[]; hasLockedRequests?: boolean }> {+ const now = Date.now();+ const items: InternalRequest[] = [];+ let hasLockedRequests = false;++ // Tracks processed request IDs to avoid duplicates (request in both `forefrontRequestIds` and `requests`).+ const seenRequestIds = new Set<string>();+ // Tracks handled request IDs from `forefrontRequestIds` to be removed.+ const handledForefrontIds = new Set<string>();++ for (const requestId of this.requestKeyIterator()) {+ // Once the requested page is filled we can stop — unless the caller asked us to detect locked+ // requests and we have not yet seen one, in which case we must keep scanning to find them.+ if (items.length >= limit && (!detectLockedRequests || hasLockedRequests)) {+ break;+ }++ if (seenRequestIds.has(requestId)) {+ continue;+ }++ seenRequestIds.add(requestId);++ const storageEntry = this.requests.get(requestId)!;++ // Cheap rejection of permanently-handled requests using the cached `orderNo` (handled is a+ // terminal state, so the cached value can be trusted without re-reading from storage).+ if (storageEntry.orderNo === null) {+ if (this.forefrontRequestIds.includes(requestId)) {+ handledForefrontIds.add(requestId);+ }+ continue;+ }++ // Re-read from storage to get fresh lock state — another process may have locked (or handled)+ // this request since we last cached it.+ const request = await storageEntry.get(true);++ // Handled in the meantime.+ if (request.orderNo === null) {+ if (this.forefrontRequestIds.includes(requestId)) {+ handledForefrontIds.add(requestId);+ }+ continue;+ }++ // Locked (in progress) by us or another process — skip until the lock expires, but remember+ // that the queue is not truly empty.+ if (isRequestLocked(request.orderNo, now)) {+ hasLockedRequests = true;+ continue;+ }++ if (items.length < limit) {+ items.push(request);+ }+ }++ this.forefrontRequestIds = this.forefrontRequestIds.filter((id) => !handledForefrontIds.has(id));++ return {+ items: items.sort((a, b) => a.orderNo! - b.orderNo!),+ hasLockedRequests: detectLockedRequests ? hasLockedRequests : undefined,+ };+ }++ async fetchNextRequest(): Promise<storage.RequestOptions | null> {+ this.updateTimestamps(false);++ await this.queueStateMutex.wait();++ try {+ const {+ items: [head],+ } = await this.listPendingHead(1);++ if (!head) {+ return null;+ }++ // Lock the request by pushing its `orderNo` beyond the lock expiry, preserving the sign so+ // its original (forefront / normal) position is restored once the lock expires. The lock is+ // persisted so other processes sharing this queue will not fetch the same request.+ const lockExpiresAt = Date.now() + this.lockSecs * 1000;+ head.orderNo = lockExpiresAt * (head.orderNo! > 0 ? 1 : -1);+ await this.requests.get(head.id)!.update(head);++ // Remember that this client owns the lock, so we can release it on process termination+ // (see `releaseOwnLocks`) instead of leaving the request stuck until the lock expires.+ this.inProgressRequestIds.add(head.id);++ return this._jsonToRequest(head.json) ?? null;+ } finally {+ this.queueStateMutex.shift();+ }+ }++ async addBatchOfRequests(+ requests: storage.RequestSchema[],+ options: storage.RequestOptions = {},+ ): Promise<storage.BatchAddRequestsResult> {+ batchRequestShapeWithoutId.parse(requests);+ requestOptionsShape.parse(options);++ // Serialize against other mutators (and the head scans in `isEmpty`/`isFinished`) so that the+ // shared `requests` map, `forefrontRequestIds` array and request counts are not corrupted by a+ // concurrent operation interleaving at one of the `await` points below.+ await this.queueStateMutex.wait();++ try {+ const result: storage.BatchAddRequestsResult = {+ processedRequests: [],+ unprocessedRequests: [],+ };++ for (const model of requests) {+ const requestModel = this._createInternalRequest(model, options.forefront);++ const existingRequestWithIdEntry = this.requests.get(requestModel.id);++ if (existingRequestWithIdEntry) {+ const existingRequestWithId = await existingRequestWithIdEntry.get();++ result.processedRequests.push({+ requestId: existingRequestWithId.id,+ uniqueKey: existingRequestWithId.uniqueKey,+ wasAlreadyHandled: existingRequestWithId.orderNo === null,+ wasAlreadyPresent: true,+ });++ continue;+ }++ const newEntry = createRequestQueueStorageImplementation({+ requestId: requestModel.id,+ storeDirectory: this.requestQueueDirectory,+ });++ await newEntry.update(requestModel);++ this.requests.set(requestModel.id, newEntry);++ if (requestModel.orderNo) {+ this.pendingRequestCount += 1;+ } else {+ this.handledRequestCount += 1;+ }++ if (options.forefront) {+ this.forefrontRequestIds.push(requestModel.id);+ }++ result.processedRequests.push({+ requestId: requestModel.id,+ uniqueKey: requestModel.uniqueKey,+ // We return wasAlreadyHandled: false even though the request may+ // have been added as handled, because that's how API behaves.+ wasAlreadyHandled: false,+ wasAlreadyPresent: false,+ });+ }++ this.updateTimestamps(true);++ return result;+ } finally {+ this.queueStateMutex.shift();+ }+ }++ async getRequest(uniqueKey: string): Promise<storage.RequestOptions | undefined> {+ s.string().parse(uniqueKey);+ this.updateTimestamps(false);+ const id = uniqueKeyToRequestId(uniqueKey);+ const json = (await this.requests.get(id)?.get())?.json;+ return this._jsonToRequest(json);+ }++ async markRequestAsHandled(request: storage.UpdateRequestSchema): Promise<storage.QueueOperationInfo | null> {+ requestShape.parse(request);+ this.updateTimestamps(false);++ // Serialize against other mutators (and the head scans in `isEmpty`/`isFinished`) so the shared+ // `requests` map, `inProgressRequestIds` set and request counts stay consistent across the+ // `await` points below.+ await this.queueStateMutex.wait();++ try {+ const id = uniqueKeyToRequestId(request.uniqueKey);++ const existingEntry = this.requests.get(id);+ const existingRequest = await existingEntry?.get();++ // The request must exist to be marked as handled. We intentionally do NOT require it to still+ // be locked: a consumer whose processing outlived the lock (slow handler, GC/event-loop pause,+ // a low `setExpectedRequestProcessingTimeSecs`) must still be able to mark its request handled,+ // otherwise the request would be handed out again forever and the queue would never finish.+ if (!existingRequest) {+ return null;+ }++ // A handled request has `orderNo === null`. Marking it again is an idempotent no-op.+ const wasAlreadyHandled = existingRequest.orderNo === null;++ const handledAt = request.handledAt ?? new Date().toISOString();+ const requestModel = this._createInternalRequest({ ...request, handledAt }, false);++ const newEntry = createRequestQueueStorageImplementation({+ requestId: id,+ storeDirectory: this.requestQueueDirectory,+ });+ await newEntry.update(requestModel);+ this.requests.set(id, newEntry);++ // The request is no longer in progress for this client.+ this.inProgressRequestIds.delete(id);++ if (!wasAlreadyHandled) {+ this.pendingRequestCount -= 1;+ this.handledRequestCount += 1;+ }++ this.updateTimestamps(true);++ return {+ requestId: id,+ wasAlreadyHandled,+ wasAlreadyPresent: true,+ };+ } finally {+ this.queueStateMutex.shift();+ }+ }++ async reclaimRequest(+ request: storage.UpdateRequestSchema,+ options: storage.RequestOptions = {},+ ): Promise<storage.QueueOperationInfo | null> {+ requestShape.parse(request);+ requestOptionsShape.parse(options);+ this.updateTimestamps(false);++ // Serialize against other mutators (and the head scans in `isEmpty`/`isFinished`) so the shared+ // `requests` map, `forefrontRequestIds` array and `inProgressRequestIds` set stay consistent+ // across the `await` points below.+ await this.queueStateMutex.wait();++ try {+ const id = uniqueKeyToRequestId(request.uniqueKey);++ const existingEntry = this.requests.get(id);+ const existingRequest = await existingEntry?.get();++ // The request must exist and not already be handled to be reclaimed. As with+ // `markRequestAsHandled`, we do NOT require it to still be locked — a consumer that failed+ // after its lock expired must still be able to return the request to the queue (e.g. to honor+ // a `forefront` reorder), rather than have the reclaim silently dropped.+ if (!existingRequest || existingRequest.orderNo === null) {+ return null;+ }++ // Reclaiming resets the `orderNo` to a fresh timestamp, releasing the lock and restoring the+ // request to the queue (at the front if `forefront`).+ const requestModel = this._createInternalRequest(request, options.forefront);++ const newEntry = createRequestQueueStorageImplementation({+ requestId: id,+ storeDirectory: this.requestQueueDirectory,+ });+ await newEntry.update(requestModel);+ this.requests.set(id, newEntry);++ // The request is no longer in progress for this client.+ this.inProgressRequestIds.delete(id);++ if (options.forefront) {+ this.forefrontRequestIds.push(id);+ }++ this.updateTimestamps(true);++ return {+ requestId: id,+ wasAlreadyHandled: false,+ wasAlreadyPresent: true,+ };+ } finally {+ this.queueStateMutex.shift();+ }+ }++ async isEmpty(): Promise<boolean> {+ this.updateTimestamps(false);++ // "Empty" means there is nothing left to fetch right now — i.e. the next `fetchNextRequest`+ // would return `null`. Requests that are currently locked (in progress) are intentionally NOT+ // counted here: they are not fetchable, so the queue is empty from a consumer's point of view.+ // Whether those in-progress requests mean crawling is not yet done is a separate question,+ // answered by `isFinished`.+ //+ // `listPendingHead` prunes `forefrontRequestIds` as it scans, so we must hold the queue-state mutex to avoid+ // racing a concurrent mutator (e.g. `addBatchOfRequests`) at its `await` points.+ await this.queueStateMutex.wait();++ try {+ const { items } = await this.listPendingHead(1);+ return items.length === 0;+ } finally {+ this.queueStateMutex.shift();+ }+ }++ async isFinished(): Promise<boolean> {+ this.updateTimestamps(false);++ // The queue is finished only when there is nothing left to fetch AND nothing currently locked+ // (in progress) by any consumer. Counting locked requests is what allows a crawler to keep+ // waiting while another consumer (possibly another process sharing this on-disk queue) still+ // holds the last requests, instead of finishing prematurely. This mirrors the Apify platform+ // shared client's `queueHasLockedRequests` signal.+ //+ // Detecting locked requests requires a full scan, hence the `detectLockedRequests` flag — unlike+ // `fetchNextRequest`/`isEmpty`, which only need the head and can stop early.+ //+ // `listPendingHead` prunes `forefrontRequestIds` as it scans, so we must hold the queue-state mutex to avoid+ // racing a concurrent mutator (e.g. `addBatchOfRequests`) at its `await` points.+ await this.queueStateMutex.wait();++ try {+ const { items, hasLockedRequests } = await this.listPendingHead(1, true);+ return items.length === 0 && !hasLockedRequests;+ } finally {+ this.queueStateMutex.shift();+ }+ }++ /**+ * Release the locks of all requests this client currently has in progress, returning them to the+ * queue so they can be fetched again immediately.+ *+ * On the Apify platform, a run's locks are released automatically when it migrates or aborts. This+ * client, however, persists fake locks via `orderNo`, so it needs to clean up after itself.+ * `FileSystemStorageClient.teardown()` calls this for every cached queue at the end of the process so that+ * a fetched-but-unhandled request is not stuck (waiting for its lock to expire) for the next consumer+ * of the same on-disk queue.+ */+ async releaseOwnLocks(): Promise<void> {+ if (this.inProgressRequestIds.size === 0) {+ return;+ }++ await this.queueStateMutex.wait();++ try {+ const now = Date.now();++ for (const id of this.inProgressRequestIds) {+ const entry = this.requests.get(id);+ const request = await entry?.get(true);++ // Skip requests that were handled or whose lock already expired/changed — we only undo+ // locks we still hold.+ if (!request || !isRequestLocked(request.orderNo, now)) {+ continue;+ }++ // Reset the lock to a fresh timestamp, preserving the sign so the request keeps its+ // original (forefront / normal) ordering once unlocked.+ request.orderNo = now * (request.orderNo! > 0 ? 1 : -1);+ await entry!.update(request);+ }++ this.inProgressRequestIds.clear();+ this.updateTimestamps(true);+ } finally {+ this.queueStateMutex.shift();+ }+ }++ toRequestQueueInfo(): storage.RequestQueueInfo {+ return {+ accessedAt: this.accessedAt,+ createdAt: this.createdAt,+ hadMultipleClients: false,+ handledRequestCount: this.handledRequestCount,+ id: this.id,+ modifiedAt: this.modifiedAt,+ name: this.name,+ pendingRequestCount: this.pendingRequestCount,+ stats: {},+ totalRequestCount: this.requests.size,+ userId: '1',+ };+ }++ private updateTimestamps(hasBeenModified: boolean) {+ this.accessedAt = new Date();++ if (hasBeenModified) {+ this.modifiedAt = new Date();+ }++ const data = {+ ...this.toRequestQueueInfo(),+ forefrontRequestIds: this.forefrontRequestIds,+ };++ scheduleBackgroundTask(+ {+ action: 'update-metadata',+ data,+ entityType: 'requestQueues',+ entityDirectory: this.requestQueueDirectory,+ id: this.name ?? this.id,+ writeMetadata: this.client.writeMetadata,+ },+ this.client.logger,+ );+ }++ private _jsonToRequest<T>(requestJson?: string): T | undefined {+ if (!requestJson) return undefined;+ const request = JSON.parse(requestJson);+ return purgeNullsFromObject(request);+ }++ private _createInternalRequest(request: storage.RequestSchema, forefront?: boolean): InternalRequest {+ const orderNo = this._calculateOrderNo(request, forefront);+ const id = uniqueKeyToRequestId(request.uniqueKey);++ if (request.id && request.id !== id) {+ throw new Error('Request ID does not match its uniqueKey.');+ }++ const json = JSON.stringify({ ...request, id });+ return {+ id,+ json,+ method: request.method,+ orderNo,+ retryCount: request.retryCount ?? 0,+ uniqueKey: request.uniqueKey,+ url: request.url,+ };+ }++ private _calculateOrderNo(request: storage.RequestSchema, forefront?: boolean) {+ if (request.handledAt) return null;++ const timestamp = Date.now();++ return forefront ? -timestamp : timestamp;+ }+}diff --git a/packages/fs-storage/src/utils.ts b/packages/fs-storage/src/utils.tsnew file mode 100644index 000000000..45686cd06--- /dev/null+++ b/packages/fs-storage/src/utils.ts@@ -0,0 +1,88 @@+import { createHash } from 'node:crypto';+import { resolve, sep } from 'node:path';++import type * as storage from '@crawlee/types';+import { s } from '@sapphire/shapeshift';++import { REQUEST_ID_LENGTH } from './consts.js';++/**+ * Resolves `segment` against `baseDirectory` and ensures the result stays within `baseDirectory`.+ * Storage names and record keys are used as filesystem path components, so a value containing `..`+ * or an absolute path could otherwise escape the intended directory.+ */+export function resolveWithinDirectory(baseDirectory: string, segment: string): string {+ const base = resolve(baseDirectory);+ const resolved = resolve(base, segment);++ if (resolved !== base && !resolved.startsWith(`${base}${sep}`)) {+ throw new Error(+ `"${segment}" is not allowed because it would resolve outside of the storage directory. ` ++ `Storage names and record keys must not contain path traversal segments ("..") or absolute paths.`,+ );+ }++ return resolved;+}++/**+ * Removes all properties with a null value+ * from the provided object.+ */+export function purgeNullsFromObject<T>(object: T): T {+ if (object && typeof object === 'object' && !Array.isArray(object)) {+ for (const [key, value] of Object.entries(object)) {+ if (value === null) Reflect.deleteProperty(object as Record<string, unknown>, key);+ }+ }++ return object;+}++/**+ * Creates a standard request ID (same as Platform).+ */+export function uniqueKeyToRequestId(uniqueKey: string): string {+ const str = createHash('sha256')+ .update(uniqueKey)+ .digest('base64')+ .replace(/(\+|\/|=)/g, '');++ return str.length > REQUEST_ID_LENGTH ? str.slice(0, REQUEST_ID_LENGTH) : str;+}++export function isBuffer(value: unknown): boolean {+ try {+ s.union([s.instance(Buffer), s.instance(ArrayBuffer), s.typedArray()]).parse(value);++ return true;+ } catch {+ return false;+ }+}++export function isStream(value: any): boolean {+ return (+ typeof value === 'object' &&+ value &&+ ['on', 'pipe'].every((key) => key in value && typeof value[key] === 'function')+ );+}++export type BackgroundHandlerReceivedMessage = BackgroundHandlerUpdateMetadataMessage;++export type BackgroundHandlerUpdateMetadataMessage =+ | MetadataUpdate<'datasets', storage.DatasetInfo>+ | MetadataUpdate<'keyValueStores', storage.KeyValueStoreInfo>+ | MetadataUpdate<'requestQueues', storage.RequestQueueInfo>;++type EntityType = 'datasets' | 'keyValueStores' | 'requestQueues';++interface MetadataUpdate<Type extends EntityType, DataType> {+ entityType: Type;+ id: string;+ action: 'update-metadata';+ entityDirectory: string;+ data: DataType;+ writeMetadata: boolean;+}diff --git a/packages/memory-storage/test/__shared__.ts b/packages/fs-storage/test/__shared__.tssimilarity index 100%rename from packages/memory-storage/test/__shared__.tsrename to packages/fs-storage/test/__shared__.tsdiff --git a/packages/fs-storage/test/async-iteration.test.ts b/packages/fs-storage/test/async-iteration.test.tsnew file mode 100644index 000000000..c37d45470--- /dev/null+++ b/packages/fs-storage/test/async-iteration.test.ts@@ -0,0 +1,114 @@+import { rm } from 'node:fs/promises';+import path from 'node:path';++import { FileSystemStorageClient } from '@crawlee/fs-storage';+import type { DatasetClient, KeyValueStoreClient } from '@crawlee/types';++describe('Async iteration support', () => {+ const localDataDirectory = path.resolve(__dirname, './tmp/async-iteration');+ const storage = new FileSystemStorageClient({+ localDataDirectory,+ });++ afterAll(async () => {+ await rm(localDataDirectory, { force: true, recursive: true });+ });++ describe('Dataset.getData', () => {+ const elements = Array.from({ length: 25 }, (_, i) => ({ index: i }));+ let dataset: DatasetClient<{ index: number }>;++ beforeAll(async () => {+ dataset = (await storage.createDatasetClient({ name: 'async-iteration-dataset' })) as DatasetClient<{+ index: number;+ }>;+ await dataset.pushData(elements);+ });++ test('getData returns a paginated result', async () => {+ const result = await dataset.getData({ limit: 10 });++ expect(result.items).toHaveLength(10);+ expect(result.total).toBe(25);+ expect(result.offset).toBe(0);+ expect(result.items).toStrictEqual(elements.slice(0, 10));+ });++ test('respects limit option', async () => {+ const result = await dataset.getData({ limit: 10 });++ expect(result.items).toHaveLength(10);+ expect(result.items).toStrictEqual(elements.slice(0, 10));+ });++ test('respects offset option', async () => {+ const result = await dataset.getData({ offset: 5 });++ expect(result.items).toHaveLength(20);+ expect(result.items).toStrictEqual(elements.slice(5));+ });++ test('respects both offset and limit options', async () => {+ const result = await dataset.getData({ offset: 5, limit: 10 });++ expect(result.items).toHaveLength(10);+ expect(result.items).toStrictEqual(elements.slice(5, 15));+ });++ test('respects desc option', async () => {+ const result = await dataset.getData({ desc: true, limit: 5 });++ expect(result.items).toHaveLength(5);+ expect(result.items).toStrictEqual(elements.slice().reverse().slice(0, 5));+ });+ });++ describe('KeyValueStore.listKeys', () => {+ const keys = Array.from({ length: 25 }, (_, i) => `key-${String(i).padStart(2, '0')}`);+ let kvStore: KeyValueStoreClient;++ beforeAll(async () => {+ kvStore = await storage.createKeyValueStoreClient({ name: 'async-iteration-kvs' });++ for (const key of keys) {+ await kvStore.setValue({ key, value: { data: key } });+ }+ });++ test('returns all keys', async () => {+ const items = await kvStore.listKeys();++ expect(items).toHaveLength(25);+ expect(items.map((i) => i.key)).toStrictEqual(keys);+ });++ test('respects prefix option', async () => {+ // Only keys starting with 'key-0' (key-00 to key-09)+ const items = await kvStore.listKeys({ prefix: 'key-0' });++ expect(items).toHaveLength(10);+ expect(items.map((i) => i.key)).toStrictEqual(keys.slice(0, 10));+ });++ test('respects exclusiveStartKey option', async () => {+ const items = await kvStore.listKeys({ exclusiveStartKey: 'key-09' });++ expect(items).toHaveLength(15);+ expect(items.map((i) => i.key)).toStrictEqual(keys.slice(10));+ });++ test('respects limit option', async () => {+ const items = await kvStore.listKeys({ limit: 5 });++ expect(items).toHaveLength(5);+ expect(items.map((i) => i.key)).toStrictEqual(keys.slice(0, 5));+ });++ test('respects exclusiveStartKey and limit together', async () => {+ const items = await kvStore.listKeys({ exclusiveStartKey: 'key-04', limit: 5 });++ expect(items).toHaveLength(5);+ expect(items.map((i) => i.key)).toStrictEqual(keys.slice(5, 10));+ });+ });+});diff --git a/packages/memory-storage/test/fs-fallback.test.ts b/packages/fs-storage/test/fs-fallback.test.tssimilarity index 97%rename from packages/memory-storage/test/fs-fallback.test.tsrename to packages/fs-storage/test/fs-fallback.test.tsindex 21c4f6434..908dab207 100644--- a/packages/memory-storage/test/fs-fallback.test.ts+++ b/packages/fs-storage/test/fs-fallback.test.ts@@ -2,13 +2,13 @@ import { randomUUID } from 'node:crypto';import { rm, writeFile } from 'node:fs/promises';import { resolve } from 'node:path';-import { MemoryStorage } from '@crawlee/memory-storage';+import { FileSystemStorageClient } from '@crawlee/fs-storage';import type { KeyValueStoreRecord } from '@crawlee/types';import { ensureDir } from 'fs-extra/esm';describe('fallback to fs for reading', () => {const tmpLocation = resolve(import.meta.dirname, './tmp/fs-fallback');- const storage = new MemoryStorage({+ const storage = new FileSystemStorageClient({localDataDirectory: tmpLocation,});diff --git a/packages/fs-storage/test/key-value-store/stream.test.ts b/packages/fs-storage/test/key-value-store/stream.test.tsnew file mode 100644index 000000000..0b4ff7709--- /dev/null+++ b/packages/fs-storage/test/key-value-store/stream.test.ts@@ -0,0 +1,20 @@+import { Readable } from 'node:stream';++import { FileSystemStorageClient } from '@crawlee/fs-storage';++describe('KeyValueStore should drain streams when setting records', () => {+ const storage = new FileSystemStorageClient({});++ const fsStream = Readable.from([Buffer.from('hello'), Buffer.from('world')]);++ test('should drain stream', async () => {+ const defaultStore = await storage.createKeyValueStoreClient({ name: 'default' });++ await defaultStore.setValue({ key: 'streamz', value: fsStream, contentType: 'text/plain' });++ expect(fsStream.destroyed).toBeTruthy();++ const record = await defaultStore.getValue('streamz');+ expect(record!.value.toString('utf8')).toEqual('helloworld');+ });+});diff --git a/packages/memory-storage/test/key-value-store/with-extension.test.ts b/packages/fs-storage/test/key-value-store/with-extension.test.tssimilarity index 95%rename from packages/memory-storage/test/key-value-store/with-extension.test.tsrename to packages/fs-storage/test/key-value-store/with-extension.test.tsindex ed29d2ce9..30f0061fc 100644--- a/packages/memory-storage/test/key-value-store/with-extension.test.ts+++ b/packages/fs-storage/test/key-value-store/with-extension.test.ts@@ -13,7 +13,6 @@ describe('KeyValueStore should append extension only when needed', () => {test('should append extension when needed (jpg)', async () => {const testDir = resolve('tmp', 'test_no_extension');const storage = createKeyValueStorageImplementation({- persistStorage: true,storeDirectory: testDir,writeMetadata: true,});@@ -31,7 +30,6 @@ describe('KeyValueStore should append extension only when needed', () => {test('should append extension when needed (html)', async () => {const testDir = resolve('tmp', 'test_no_extension');const storage = createKeyValueStorageImplementation({- persistStorage: true,storeDirectory: testDir,writeMetadata: true,});@@ -49,7 +47,6 @@ describe('KeyValueStore should append extension only when needed', () => {test('should not append extension when already available', async () => {const testDir = resolve('tmp', 'test_extension');const storage = createKeyValueStorageImplementation({- persistStorage: true,storeDirectory: testDir,writeMetadata: true,});@@ -67,7 +64,6 @@ describe('KeyValueStore should append extension only when needed', () => {test('should not append extension when already available', async () => {const testDir = resolve('tmp', 'test_extension');const storage = createKeyValueStorageImplementation({- persistStorage: true,storeDirectory: testDir,writeMetadata: true,});diff --git a/packages/fs-storage/test/no-crash-on-big-buffers.test.ts b/packages/fs-storage/test/no-crash-on-big-buffers.test.tsnew file mode 100644index 000000000..0ecdd4bdf--- /dev/null+++ b/packages/fs-storage/test/no-crash-on-big-buffers.test.ts@@ -0,0 +1,43 @@+// https://github.com/apify/crawlee/issues/1732+// https://github.com/apify/crawlee/issues/1710++import { rm } from 'node:fs/promises';+import { resolve } from 'node:path';++import { FileSystemStorageClient } from '@crawlee/fs-storage';+import type { KeyValueStoreClient } from '@crawlee/types';++describe('FileSystemStorageClient should not crash when saving a big buffer', () => {+ const tmpLocation = resolve(import.meta.dirname, './tmp/no-buffer-crash');+ const storage = new FileSystemStorageClient({+ localDataDirectory: tmpLocation,+ });++ let store: KeyValueStoreClient;++ beforeAll(async () => {+ store = await storage.createKeyValueStoreClient();+ });++ afterAll(async () => {+ await rm(tmpLocation, { force: true, recursive: true });+ });++ test('should not crash when saving a big buffer', async () => {+ let zip: Buffer;++ if (process.env.CRAWLEE_DIFFICULT_TESTS) {+ const numbers = Array.from([...Array(18_100_000).keys()].map((i) => i * 3_000_000));++ zip = Buffer.from([...numbers]);+ } else {+ zip = Buffer.from([...Array(100_000)].map((i) => i * 8));+ }++ try {+ await store.setValue({ key: 'owo.zip', value: zip });+ } catch (err) {+ expect(err).not.toBeDefined();+ }+ });+});diff --git a/packages/fs-storage/test/request-queue/dangling-forefront-on-reload.test.ts b/packages/fs-storage/test/request-queue/dangling-forefront-on-reload.test.tsnew file mode 100644index 000000000..0dec65342--- /dev/null+++ b/packages/fs-storage/test/request-queue/dangling-forefront-on-reload.test.ts@@ -0,0 +1,86 @@+import { randomUUID } from 'node:crypto';+import { rm, writeFile } from 'node:fs/promises';+import { resolve } from 'node:path';++import { FileSystemStorageClient } from '@crawlee/fs-storage';+import type { InternalRequest } from '@crawlee/fs-storage/src/resource-clients/request-queue';+import type { RequestSchema } from '@crawlee/types';+import { ensureDir } from 'fs-extra/esm';++/**+ * On reload, `forefrontRequestIds` is restored verbatim from the persisted metadata, while the+ * `requests` map is rebuilt only from request files actually found and parseable on disk. If a+ * persisted forefront id has no backing request file (deleted, never written, or corrupt JSON), it+ * would dangle: a later head scan resolves it to a missing request and dereferences `undefined`.+ * The reload must drop such ids so head scans stay safe.+ */+describe('Request queue reload drops dangling forefront ids without a backing request file', () => {+ const tmpLocation = resolve(import.meta.dirname, './tmp/req-queue-dangling-forefront');++ const writeMetadata = async (storage: FileSystemStorageClient, forefrontRequestIds: string[]) => {+ await ensureDir(resolve(storage.requestQueuesDirectory, 'default'));+ await writeFile(+ resolve(storage.requestQueuesDirectory, 'default/__metadata__.json'),+ JSON.stringify({+ id: randomUUID(),+ name: 'default',+ createdAt: new Date(2022, 0, 1),+ accessedAt: new Date(2022, 0, 1),+ modifiedAt: new Date(2022, 0, 1),+ pendingRequestCount: 1,+ handledRequestCount: 0,+ forefrontRequestIds,+ }),+ );+ };++ const writeValidRequest = async (storage: FileSystemStorageClient, id: string, url: string, uniqueKey: string) => {+ await writeFile(+ resolve(storage.requestQueuesDirectory, `default/${id}.json`),+ JSON.stringify({+ id,+ orderNo: -1,+ url,+ uniqueKey,+ method: 'GET',+ retryCount: 0,+ json: JSON.stringify({ id, url, uniqueKey } satisfies RequestSchema),+ } satisfies InternalRequest),+ );+ };++ afterEach(async () => {+ await rm(tmpLocation, { force: true, recursive: true });+ });++ test('a forefront id with no request file on disk does not crash a later head scan', async () => {+ const storage = new FileSystemStorageClient({ localDataDirectory: tmpLocation });++ // The metadata references a forefront id 'missing' for which no request file exists on disk.+ await writeMetadata(storage, ['missing']);++ const queue = await storage.createRequestQueueClient({ name: 'default' });++ // None of these head scans must throw, and an empty/finished queue must be reported.+ await expect(queue.isEmpty()).resolves.toBe(true);+ await expect(queue.isFinished()).resolves.toBe(true);+ await expect(queue.fetchNextRequest()).resolves.toBeNull();+ });++ test('a valid forefront request is still served while a dangling sibling id is dropped', async () => {+ const storage = new FileSystemStorageClient({ localDataDirectory: tmpLocation });++ // Two forefront ids in metadata: one backed by a real file, one dangling.+ await writeMetadata(storage, ['missing', '123']);+ await writeValidRequest(storage, '123', 'http://example.com', 'owo');++ const queue = await storage.createRequestQueueClient({ name: 'default' });++ // The dangling id is dropped; the valid forefront request is still fetchable.+ const first = await queue.fetchNextRequest();+ expect(first).not.toBeNull();+ expect(first!.url).toEqual('http://example.com');++ expect(await queue.fetchNextRequest()).toBeNull();+ });+});diff --git a/packages/fs-storage/test/request-queue/forefront.test.ts b/packages/fs-storage/test/request-queue/forefront.test.tsnew file mode 100644index 000000000..b95ed944d--- /dev/null+++ b/packages/fs-storage/test/request-queue/forefront.test.ts@@ -0,0 +1,266 @@+import { rm } from 'node:fs/promises';+import { resolve } from 'node:path';+import { setTimeout as sleep } from 'node:timers/promises';++import { FileSystemStorageClient } from '@crawlee/fs-storage';+import type { RequestQueueClient } from '@crawlee/types';++/**+ * Drains the queue via `fetchNextRequest`, marking each request as handled, and returns the+ * pathnames in the order they were served.+ */+async function fetchOrder(client: RequestQueueClient): Promise<string[]> {+ const order: string[] = [];++ for (let request = await client.fetchNextRequest(); request !== null; request = await client.fetchNextRequest()) {+ order.push(new URL(request.url).pathname);+ await client.markRequestAsHandled({ ...request, id: request.id! });+ }++ return order;+}++describe('RequestQueue respects `forefront` when fetching requests', () => {+ const storage = new FileSystemStorageClient({});++ let requestQueue: RequestQueueClient;++ beforeEach(async () => {+ requestQueue = await storage.createRequestQueueClient({ name: 'forefront' });+ });++ afterEach(async () => {+ await requestQueue.drop();+ });++ test('requests without `forefront` respect sequential order', async () => {+ await requestQueue.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }]);+ // Waiting a few ms is required since we use Date.now() to compute orderNo+ await sleep(2);+ await requestQueue.addBatchOfRequests([{ url: 'http://example.com/2', uniqueKey: '2' }]);++ expect(await fetchOrder(requestQueue)).toEqual(['/1', '/2']);+ });++ test('`forefront` requests are prioritized', async () => {+ await requestQueue.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }]);+ // Waiting a few ms is required since we use Date.now() to compute orderNo+ await sleep(2);+ await requestQueue.addBatchOfRequests([{ url: 'http://example.com/2', uniqueKey: '2' }], { forefront: true });++ expect(await fetchOrder(requestQueue)).toEqual(['/2', '/1']);+ });++ test('global `forefront` ordering is preserved across several inserts', async () => {+ await requestQueue.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }]);+ await sleep(2);+ await requestQueue.addBatchOfRequests([{ url: 'http://example.com/2', uniqueKey: '2' }], { forefront: true });+ await sleep(2);+ await requestQueue.addBatchOfRequests([{ url: 'http://example.com/3', uniqueKey: '3' }], { forefront: true });++ expect(await fetchOrder(requestQueue)).toEqual(['/3', '/2', '/1']);+ });++ test('`addBatchOfRequests` respects `forefront`', async () => {+ await requestQueue.addBatchOfRequests([{ url: 'http://example.com/3', uniqueKey: '3' }]);++ await sleep(2);++ await requestQueue.addBatchOfRequests(+ [+ { url: 'http://example.com/1', uniqueKey: '1' },+ { url: 'http://example.com/2', uniqueKey: '2' },+ ],+ { forefront: true },+ );++ const order = await fetchOrder(requestQueue);+ expect(order).toHaveLength(3);+ // Both forefront requests come before the original; their relative order is arbitrary.+ expect(order[2]).toEqual('/3');+ expect([+ ['/2', '/1', '/3'],+ ['/1', '/2', '/3'],+ ]).toContainEqual(order);+ });++ test('a reclaimed request is served again', async () => {+ await requestQueue.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }]);++ const first = await requestQueue.fetchNextRequest();+ expect(first!.url).toEqual('http://example.com/1');++ // Reclaiming a fetched (in-progress) request returns it to the queue.+ await requestQueue.reclaimRequest({ ...first!, id: first!.id! });++ const second = await requestQueue.fetchNextRequest();+ expect(second!.url).toEqual('http://example.com/1');+ });++ test('a reclaimed `forefront` request jumps to the front', async () => {+ await requestQueue.addBatchOfRequests([+ { url: 'http://example.com/1', uniqueKey: '1' },+ { url: 'http://example.com/2', uniqueKey: '2' },+ ]);++ const first = await requestQueue.fetchNextRequest();+ expect(first!.url).toEqual('http://example.com/1');++ await requestQueue.reclaimRequest({ ...first!, id: first!.id! }, { forefront: true });++ const next = await requestQueue.fetchNextRequest();+ expect(next!.url).toEqual('http://example.com/1');+ });++ test('handling all requests empties the queue', async () => {+ await requestQueue.addBatchOfRequests([+ { url: 'http://example.com/1', uniqueKey: '1' },+ { url: 'http://example.com/2', uniqueKey: '2' },+ { url: 'http://example.com/3', uniqueKey: '3' },+ ]);++ expect(await requestQueue.isEmpty()).toBe(false);++ await fetchOrder(requestQueue);++ expect(await requestQueue.isEmpty()).toBe(true);+ expect(await requestQueue.fetchNextRequest()).toBeNull();+ });++ test('a fetched (locked) request leaves the queue empty but unfinished until it is handled', async () => {+ await requestQueue.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }]);++ const request = await requestQueue.fetchNextRequest();+ expect(request).not.toBeNull();++ // The request is locked (in progress), not handled. There is nothing left to fetch, so the+ // queue is empty — but it is not finished. The "not finished" signal is what stops a crawler+ // from shutting down while a request is still being processed by some consumer.+ expect(await requestQueue.isEmpty()).toBe(true);+ expect(await requestQueue.isFinished()).toBe(false);++ await requestQueue.markRequestAsHandled({ ...request!, id: request!.id! });+ expect(await requestQueue.isEmpty()).toBe(true);+ expect(await requestQueue.isFinished()).toBe(true);+ });+});++describe('RequestQueue locks fetched requests', () => {+ const storage = new FileSystemStorageClient({});++ let requestQueue: RequestQueueClient;++ beforeEach(async () => {+ requestQueue = await storage.createRequestQueueClient({ name: 'locking' });+ });++ afterEach(async () => {+ await requestQueue.drop();+ });++ test('a fetched request becomes available again after its lock expires', async () => {+ vitest.useFakeTimers();++ try {+ await requestQueue.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }]);++ const first = await requestQueue.fetchNextRequest();+ expect(first!.uniqueKey).toBe('1');++ // While locked, the request is not handed out again.+ expect(await requestQueue.fetchNextRequest()).toBeNull();++ // After the lock expires (default 3 minutes), the request is fetchable again — this is what+ // prevents a crashed consumer from blocking its requests forever.+ vitest.advanceTimersByTime(3 * 60 * 1000 + 1000);++ const retried = await requestQueue.fetchNextRequest();+ expect(retried!.uniqueKey).toBe('1');+ } finally {+ vitest.useRealTimers();+ }+ });+});++describe('RequestQueue locking is visible across clients sharing on-disk storage', () => {+ const tmpLocation = resolve(import.meta.dirname, './tmp/req-queue-cross-process');+ // Two independent storage instances over the same directory emulate two separate processes.+ const storageA = new FileSystemStorageClient({ localDataDirectory: tmpLocation });+ const storageB = new FileSystemStorageClient({ localDataDirectory: tmpLocation });++ afterAll(async () => {+ await rm(tmpLocation, { force: true, recursive: true });+ });++ test('two clients on the same queue never fetch the same request', async () => {+ const clientA = await storageA.createRequestQueueClient({ name: 'shared' });+ await clientA.addBatchOfRequests([+ { url: 'http://example.com/1', uniqueKey: '1' },+ { url: 'http://example.com/2', uniqueKey: '2' },+ ]);++ const clientB = await storageB.createRequestQueueClient({ name: 'shared' });++ const fromA = await clientA.fetchNextRequest();+ const fromB = await clientB.fetchNextRequest();++ expect(fromA).not.toBeNull();+ expect(fromB).not.toBeNull();+ // The lock written by one client is observed by the other, so they get distinct requests.+ expect(fromA!.uniqueKey).not.toBe(fromB!.uniqueKey);++ // Both requests are now locked, so neither client can fetch anything more.+ expect(await clientA.fetchNextRequest()).toBeNull();+ expect(await clientB.fetchNextRequest()).toBeNull();++ await clientA.drop();+ });++ test('a client does not report the queue finished while another client holds the last request', async () => {+ const clientA = await storageA.createRequestQueueClient({ name: 'shared-is-empty' });+ await clientA.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }]);++ const clientB = await storageB.createRequestQueueClient({ name: 'shared-is-empty' });++ // Client A fetches (and thus locks) the only request.+ const fromA = await clientA.fetchNextRequest();+ expect(fromA).not.toBeNull();++ // Client B has nothing it can fetch right now, so from its point of view the queue is empty...+ expect(await clientB.fetchNextRequest()).toBeNull();+ expect(await clientB.isEmpty()).toBe(true);+ // ...but the request still exists and is merely locked by A, so B must NOT consider the queue+ // finished — otherwise the crawler driving B could shut down while A is still processing.+ expect(await clientB.isFinished()).toBe(false);++ // Once A handles the request, it is gone for good and B sees a finished queue.+ await clientA.markRequestAsHandled({ ...fromA!, id: fromA!.id! });+ expect(await clientB.isEmpty()).toBe(true);+ expect(await clientB.isFinished()).toBe(true);++ await clientA.drop();+ });++ test('teardown releases this client locks so another client can fetch immediately', async () => {+ const clientA = await storageA.createRequestQueueClient({ name: 'shared-teardown' });+ await clientA.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }]);++ // Client A fetches (locks) the request, then the process tears down without handling it.+ const fromA = await clientA.fetchNextRequest();+ expect(fromA).not.toBeNull();++ const clientB = await storageB.createRequestQueueClient({ name: 'shared-teardown' });+ // While A holds the lock, B cannot fetch the request.+ expect(await clientB.fetchNextRequest()).toBeNull();++ // Tearing down A's storage releases its locks (instead of leaving them until the 3-minute+ // expiry), so B can pick the request up right away.+ await storageA.teardown();++ const fromB = await clientB.fetchNextRequest();+ expect(fromB).not.toBeNull();+ expect(fromB!.uniqueKey).toBe('1');++ await clientB.drop();+ });+});diff --git a/packages/fs-storage/test/request-queue/handledRequestCount-should-update.test.ts b/packages/fs-storage/test/request-queue/handledRequestCount-should-update.test.tsnew file mode 100644index 000000000..eff5934d0--- /dev/null+++ b/packages/fs-storage/test/request-queue/handledRequestCount-should-update.test.ts@@ -0,0 +1,51 @@+import { rm } from 'node:fs/promises';+import { resolve } from 'node:path';++import { FileSystemStorageClient } from '@crawlee/fs-storage';+import type { RequestQueueClient } from '@crawlee/types';++describe('RequestQueue handledRequestCount should update', () => {+ // Use an isolated storage directory so persisted request files from a previous run cannot leak+ // into this one (a handled request surviving on disk would be deduplicated on the next add).+ const localDataDirectory = resolve(import.meta.dirname, './tmp/handled-request-count');+ const storage = new FileSystemStorageClient({ localDataDirectory });++ let requestQueue: RequestQueueClient;++ beforeAll(async () => {+ requestQueue = await storage.createRequestQueueClient({ name: 'handledRequestCount' });+ });++ afterAll(async () => {+ await rm(localDataDirectory, { force: true, recursive: true });+ });++ test('after marking a request as handled, it should increment the handledRequestCount', async () => {+ await requestQueue.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }]);++ const request = await requestQueue.fetchNextRequest();+ expect(request).not.toBeNull();++ await requestQueue.markRequestAsHandled({+ url: 'http://example.com/1',+ uniqueKey: '1',+ id: request!.id!,+ });++ const updatedStatistics = await requestQueue.getMetadata();+ expect(updatedStatistics.handledRequestCount).toEqual(1);+ });++ test('adding an already handled request should increment the handledRequestCount', async () => {+ await requestQueue.addBatchOfRequests([+ {+ url: 'http://example.com/2',+ uniqueKey: '2',+ handledAt: new Date().toISOString(),+ },+ ]);++ const updatedStatistics = await requestQueue.getMetadata();+ expect(updatedStatistics.handledRequestCount).toEqual(2);+ });+});diff --git a/packages/memory-storage/test/request-queue/ignore-non-json-files.test.ts b/packages/fs-storage/test/request-queue/ignore-non-json-files.test.tssimilarity index 92%rename from packages/memory-storage/test/request-queue/ignore-non-json-files.test.tsrename to packages/fs-storage/test/request-queue/ignore-non-json-files.test.tsindex 8b9f54692..2525b025a 100644--- a/packages/memory-storage/test/request-queue/ignore-non-json-files.test.ts+++ b/packages/fs-storage/test/request-queue/ignore-non-json-files.test.ts@@ -2,14 +2,14 @@ import { randomUUID } from 'node:crypto';import { rm, writeFile } from 'node:fs/promises';import { resolve } from 'node:path';-import { MemoryStorage } from '@crawlee/memory-storage';-import type { InternalRequest } from '@crawlee/memory-storage/src/resource-clients/request-queue';+import { FileSystemStorageClient } from '@crawlee/fs-storage';+import type { InternalRequest } from '@crawlee/fs-storage/src/resource-clients/request-queue';import type { RequestSchema } from '@crawlee/types';import { ensureDir } from 'fs-extra/esm';describe('when falling back to fs, Request queue should ignore non-JSON files', () => {const tmpLocation = resolve(import.meta.dirname, './tmp/req-queue-ignore-non-json');- const storage = new MemoryStorage({+ const storage = new FileSystemStorageClient({localDataDirectory: tmpLocation,});diff --git a/packages/fs-storage/test/reverse-datataset-list.test.ts b/packages/fs-storage/test/reverse-datataset-list.test.tsnew file mode 100644index 000000000..dbafc72ab--- /dev/null+++ b/packages/fs-storage/test/reverse-datataset-list.test.ts@@ -0,0 +1,54 @@+import { rm } from 'node:fs/promises';+import { resolve } from 'node:path';++import { FileSystemStorageClient } from '@crawlee/fs-storage';+import type { DatasetClient } from '@crawlee/types';++const elements = Array.from({ length: 10 }, (_, i) => ({ number: i }));++describe('Dataset#getData respects the desc option', () => {+ const localDataDirectory = resolve(import.meta.dirname, './tmp/desc');+ const storage = new FileSystemStorageClient({+ localDataDirectory,+ });++ let dataset: DatasetClient;++ afterAll(async () => {+ await rm(localDataDirectory, { force: true, recursive: true });+ });++ beforeAll(async () => {+ dataset = await storage.createDatasetClient({ name: 'false' });++ await dataset.pushData(elements);+ });++ test('with desc: false', async () => {+ const result = await dataset.getData({ desc: false, limit: 5 });++ expect(result.items).toHaveLength(5);+ expect(result.items).toStrictEqual(elements.slice(0, 5));+ });++ test('with desc: true', async () => {+ const result = await dataset.getData({ desc: true, limit: 5 });++ expect(result.items).toHaveLength(5);+ expect(result.items).toStrictEqual(elements.slice().reverse().slice(0, 5));+ });++ test('with desc: false and offset: 2', async () => {+ const result = await dataset.getData({ desc: false, limit: 5, offset: 2 });++ expect(result.items).toHaveLength(5);+ expect(result.items).toStrictEqual(elements.slice(2, 7));+ });++ test('with desc: true and offset: 2', async () => {+ const result = await dataset.getData({ desc: true, limit: 5, offset: 2 });++ expect(result.items).toHaveLength(5);+ expect(result.items).toStrictEqual(elements.slice().reverse().slice(2, 7));+ });+});diff --git a/packages/fs-storage/test/tsconfig.json b/packages/fs-storage/test/tsconfig.jsonnew file mode 100644index 000000000..eb8cbab58--- /dev/null+++ b/packages/fs-storage/test/tsconfig.json@@ -0,0 +1,7 @@+{+ "extends": "../../../tsconfig.json",+ "include": ["**/*", "../../**/*"],+ "compilerOptions": {+ "types": ["vitest/globals"]+ }+}diff --git a/packages/memory-storage/test/write-metadata.test.ts b/packages/fs-storage/test/write-metadata.test.tssimilarity index 95%rename from packages/memory-storage/test/write-metadata.test.tsrename to packages/fs-storage/test/write-metadata.test.tsindex 80a6686ac..d2a357f4f 100644--- a/packages/memory-storage/test/write-metadata.test.ts+++ b/packages/fs-storage/test/write-metadata.test.ts@@ -1,7 +1,7 @@import { readdir, rm } from 'node:fs/promises';import { resolve } from 'node:path';-import { MemoryStorage } from '@crawlee/memory-storage';+import { FileSystemStorageClient } from '@crawlee/fs-storage';import { waitTillWrittenToDisk } from './__shared__.js';@@ -14,7 +14,7 @@ describe('writeMetadata option', () => {describe('when false', () => {const localDataDirectory = resolve(tmpLocation, './no-metadata');- const storage = new MemoryStorage({+ const storage = new FileSystemStorageClient({localDataDirectory,writeMetadata: false,});@@ -44,7 +44,7 @@ describe('writeMetadata option', () => {describe('when true', () => {const localDataDirectory = resolve(tmpLocation, './metadata');- const storage = new MemoryStorage({+ const storage = new FileSystemStorageClient({localDataDirectory,writeMetadata: true,});diff --git a/packages/fs-storage/tsconfig.build.json b/packages/fs-storage/tsconfig.build.jsonnew file mode 100644index 000000000..5f63b6d3d--- /dev/null+++ b/packages/fs-storage/tsconfig.build.json@@ -0,0 +1,8 @@+{+ "extends": "../../tsconfig.build.json",+ "compilerOptions": {+ "outDir": "./dist",+ "rootDir": "./src"+ },+ "include": ["src/**/*"]+}diff --git a/packages/fs-storage/tsconfig.json b/packages/fs-storage/tsconfig.jsonnew file mode 100644index 000000000..66bb87a91--- /dev/null+++ b/packages/fs-storage/tsconfig.json@@ -0,0 +1,4 @@+{+ "extends": "../../tsconfig.json",+ "include": ["src/**/*"]+}diff --git a/packages/memory-storage/package.json b/packages/memory-storage/package.jsonindex 0180cef88..6bcc98532 100644--- a/packages/memory-storage/package.json+++ b/packages/memory-storage/package.json@@ -46,11 +46,8 @@"@sapphire/async-queue": "^1.5.5","@sapphire/shapeshift": "^4.0.0","content-type": "^1.0.5",- "fs-extra": "^11.3.0","json5": "^2.2.3","mime-types": "^3.0.1",- "p-limit": "^6.2.0",- "proper-lockfile": "^4.1.2","tslib": "^2.8.1"}}diff --git a/packages/memory-storage/src/consts.ts b/packages/memory-storage/src/consts.tsindex 3d7bdc532..db8b29760 100644--- a/packages/memory-storage/src/consts.ts+++ b/packages/memory-storage/src/consts.ts@@ -2,17 +2,3 @@* Length of id property of a Request instance in characters.*/export const REQUEST_ID_LENGTH = 15;--/**- * Types of all emulated storages (currently used for warning messages only).- */-export enum StorageTypes {- RequestQueue = 'Request queue',- KeyValueStore = 'Key-value store',- Dataset = 'Dataset',-}--/**- * Except in dataset items, the default limit for API results is 1000.- */-export const DEFAULT_API_PARAM_LIMIT = 1000;diff --git a/packages/memory-storage/src/fs/dataset/memory.ts b/packages/memory-storage/src/fs/dataset/memory.tsdeleted file mode 100644index c0a7e6f47..000000000--- a/packages/memory-storage/src/fs/dataset/memory.ts+++ /dev/null@@ -1,17 +0,0 @@-import type { StorageImplementation } from '../common.js';--export class DatasetMemoryEntry<Data> implements StorageImplementation<Data> {- private data!: Data;-- async get() {- return this.data;- }-- update(data: Data) {- this.data = data;- }-- delete() {- // No-op- }-}diff --git a/packages/memory-storage/src/fs/key-value-store/memory.ts b/packages/memory-storage/src/fs/key-value-store/memory.tsdeleted file mode 100644index 0ac0e66ef..000000000--- a/packages/memory-storage/src/fs/key-value-store/memory.ts+++ /dev/null@@ -1,18 +0,0 @@-import type { InternalKeyRecord } from '../../resource-clients/key-value-store.js';-import type { StorageImplementation } from '../common.js';--export class KeyValueMemoryEntry implements StorageImplementation<InternalKeyRecord> {- private data!: InternalKeyRecord;-- async get() {- return this.data;- }-- update(data: InternalKeyRecord) {- this.data = data;- }-- delete() {- // No-op- }-}diff --git a/packages/memory-storage/src/fs/request-queue/memory.ts b/packages/memory-storage/src/fs/request-queue/memory.tsdeleted file mode 100644index 9399c2ed4..000000000--- a/packages/memory-storage/src/fs/request-queue/memory.ts+++ /dev/null@@ -1,21 +0,0 @@-import type { InternalRequest } from '../../resource-clients/request-queue.js';-import type { StorageImplementation } from '../common.js';--export class RequestQueueMemoryEntry implements StorageImplementation<InternalRequest> {- private data!: InternalRequest;-- public orderNo?: number | null;-- async get() {- return this.data;- }-- update(data: InternalRequest) {- this.data = data;- this.orderNo = data.orderNo;- }-- delete() {- // No-op- }-}diff --git a/packages/memory-storage/src/memory-storage.ts b/packages/memory-storage/src/memory-storage.tsindex 3fe71a3e9..c6156d01c 100644--- a/packages/memory-storage/src/memory-storage.ts+++ b/packages/memory-storage/src/memory-storage.ts@@ -1,279 +1,154 @@-/* eslint-disable import/no-duplicates */-import { access, readdir, rm } from 'node:fs/promises';-import { resolve } from 'node:path';+import { randomUUID } from 'node:crypto';import type * as storage from '@crawlee/types';import type { CrawleeLogger } from '@crawlee/types';import { s } from '@sapphire/shapeshift';-import { ensureDirSync, move, moveSync, pathExistsSync } from 'fs-extra/esm';-import { promiseMap, scheduleBackgroundTask } from './background-handler/index.js';-import {- findOrCacheDatasetByPossibleId,- findOrCacheKeyValueStoreByPossibleId,- findRequestQueueByPossibleId,-} from './cache-helpers.js';import { DatasetClient } from './resource-clients/dataset.js';import { KeyValueStoreClient } from './resource-clients/key-value-store.js';import { RequestQueueClient } from './resource-clients/request-queue.js';export interface MemoryStorageOptions {/**- * Path to directory where the data will also be saved.- * @default process.env.CRAWLEE_STORAGE_DIR ?? './storage'- */- localDataDirectory?: string;-- /**- * Whether to also write optional metadata files when storing to disk.- * @default process.env.DEBUG?.includes('*') ?? process.env.DEBUG?.includes('crawlee:memory-storage') ?? false- */- writeMetadata?: boolean;-- /**- * Whether the memory storage should also write its stored content to the disk.- *- * You can also disable this by setting the `CRAWLEE_PERSIST_STORAGE` environment variable to `false`.- * @default true- */- persistStorage?: boolean;-- /**- * Optional logger for MemoryStorage warnings.+ * Optional logger for MemoryStorageClient warnings.*/logger?: CrawleeLogger;}-export class MemoryStorage implements storage.StorageClient {- readonly localDataDirectory: string;- readonly datasetsDirectory: string;- readonly keyValueStoresDirectory: string;- readonly requestQueuesDirectory: string;- readonly writeMetadata: boolean;- readonly persistStorage: boolean;+export class MemoryStorageClient implements storage.StorageClient {readonly logger?: CrawleeLogger;+ /**+ * Unique per-instance cache partition key. Mirrors the way `FileSystemStorageClient` partitions its+ * cache by storage directory: two distinct `MemoryStorageClient` instances must not share cached clients.+ */+ private readonly instanceCacheKey = `MemoryStorageClient:${randomUUID()}`;+readonly keyValueStoreCache: KeyValueStoreClient[] = [];readonly datasetClientCache: DatasetClient[] = [];readonly requestQueueCache: RequestQueueClient[] = [];constructor(options: MemoryStorageOptions = {}) {- s.object({- localDataDirectory: s.string().optional(),- writeMetadata: s.boolean().optional(),- persistStorage: s.boolean().optional(),- }).parse(options);-this.logger = options.logger;-- // v3.0.0 used `crawlee_storage` as the default, we changed this in v3.0.1 to just `storage`,- // this function handles it without making BC breaks - it respects existing `crawlee_storage`- // directories, and uses the `storage` only if it's not there.- const defaultStorageDir = () => {- if (pathExistsSync(resolve('./crawlee_storage'))) {- return './crawlee_storage';- }-- return './storage';- };-- this.localDataDirectory = options.localDataDirectory ?? process.env.CRAWLEE_STORAGE_DIR ?? defaultStorageDir();- this.datasetsDirectory = resolve(this.localDataDirectory, 'datasets');- this.keyValueStoresDirectory = resolve(this.localDataDirectory, 'key_value_stores');- this.requestQueuesDirectory = resolve(this.localDataDirectory, 'request_queues');- this.writeMetadata =- options.writeMetadata ??- process.env.DEBUG?.includes('*') ??- process.env.DEBUG?.includes('crawlee:memory-storage') ??- false;- this.persistStorage =- options.persistStorage ??- (process.env.CRAWLEE_PERSIST_STORAGE- ? !['false', '0', ''].includes(process.env.CRAWLEE_PERSIST_STORAGE!)- : true);}/**- * Return a cache key that includes the resolved storage directory, so that- * two `MemoryStorage` instances pointing at different directories get separate- * cache partitions. Mirrors crawlee-python's `FileSystemStorageClient` which- * includes `configuration.storage_dir` in its cache key.+ * Return a per-instance unique cache key so that distinct `MemoryStorageClient` instances get separate+ * cache partitions in the storage-client cache.*/getStorageClientCacheKey(): string {- return `MemoryStorage:${resolve(this.localDataDirectory)}`;+ return this.instanceCacheKey;}private static resolveStorageKey(options: { id?: string; name?: string; alias?: string }): {isAlias: boolean;- directoryKey: string | undefined;+ cacheKey: string | undefined;} {const isAlias = 'alias' in options && !!options.alias;const rawKey = isAlias ? options.alias : (options.name ?? options.id);// Normalize the internal __default__ alias to the user-facing 'default' name.- const directoryKey = rawKey === '__default__' ? 'default' : rawKey;- return { isAlias, directoryKey };+ const cacheKey = rawKey === '__default__' ? 'default' : rawKey;+ return { isAlias, cacheKey };}async createDatasetClient(options: storage.CreateDatasetClientOptions = {}): Promise<storage.DatasetClient> {- const { isAlias, directoryKey } = MemoryStorage.resolveStorageKey(options);-- if (directoryKey) {- const found = await findOrCacheDatasetByPossibleId(this, directoryKey);+ const { isAlias, cacheKey } = MemoryStorageClient.resolveStorageKey(options);++ if (cacheKey) {+ const found = this.datasetClientCache.find(+ (store) =>+ store.id === cacheKey ||+ store.name?.toLowerCase() === cacheKey.toLowerCase() ||+ store.cacheKey.toLowerCase() === cacheKey.toLowerCase(),+ );if (found) {return found;}}const newStore = new DatasetClient({- name: isAlias ? undefined : directoryKey,- directoryName: directoryKey,- baseStorageDirectory: this.datasetsDirectory,+ name: isAlias ? undefined : cacheKey,+ cacheKey,client: this,});this.datasetClientCache.push(newStore);- // Schedule the worker to write to the disk- const datasetInfo = newStore.toDatasetInfo();-- scheduleBackgroundTask(- {- action: 'update-metadata',- entityType: 'datasets',- entityDirectory: newStore.datasetDirectory,- id: datasetInfo.name ?? datasetInfo.id,- data: datasetInfo,- writeMetadata: this.writeMetadata,- persistStorage: this.persistStorage,- },- this.logger,- );-return newStore;}async createKeyValueStoreClient(options: storage.CreateKeyValueStoreClientOptions = {},): Promise<storage.KeyValueStoreClient> {- const { isAlias, directoryKey } = MemoryStorage.resolveStorageKey(options);-- if (directoryKey) {- const found = await findOrCacheKeyValueStoreByPossibleId(this, directoryKey);+ const { isAlias, cacheKey } = MemoryStorageClient.resolveStorageKey(options);++ if (cacheKey) {+ const found = this.keyValueStoreCache.find(+ (store) =>+ store.id === cacheKey ||+ store.name?.toLowerCase() === cacheKey.toLowerCase() ||+ store.cacheKey.toLowerCase() === cacheKey.toLowerCase(),+ );if (found) {return found;}}const newStore = new KeyValueStoreClient({- name: isAlias ? undefined : directoryKey,- directoryName: directoryKey,- baseStorageDirectory: this.keyValueStoresDirectory,+ name: isAlias ? undefined : cacheKey,+ cacheKey,client: this,});this.keyValueStoreCache.push(newStore);- // Schedule the worker to write to the disk- const kvStoreInfo = newStore.toKeyValueStoreInfo();-- scheduleBackgroundTask(- {- action: 'update-metadata',- entityType: 'keyValueStores',- entityDirectory: newStore.keyValueStoreDirectory,- id: kvStoreInfo.name ?? kvStoreInfo.id,- data: kvStoreInfo,- writeMetadata: this.writeMetadata,- persistStorage: this.persistStorage,- },- this.logger,- );-return newStore;}async createRequestQueueClient(options: storage.CreateRequestQueueClientOptions = {},): Promise<storage.RequestQueueClient> {- const { isAlias, directoryKey } = MemoryStorage.resolveStorageKey(options);-- if (directoryKey) {- const found = await findRequestQueueByPossibleId(this, directoryKey);+ const { isAlias, cacheKey } = MemoryStorageClient.resolveStorageKey(options);++ if (cacheKey) {+ const found = this.requestQueueCache.find(+ (queue) =>+ queue.id === cacheKey ||+ queue.name?.toLowerCase() === cacheKey.toLowerCase() ||+ queue.cacheKey.toLowerCase() === cacheKey.toLowerCase(),+ );if (found) {return found;}}const newStore = new RequestQueueClient({- name: isAlias ? undefined : directoryKey,- directoryName: directoryKey,- baseStorageDirectory: this.requestQueuesDirectory,+ name: isAlias ? undefined : cacheKey,+ cacheKey,client: this,});this.requestQueueCache.push(newStore);- // Schedule the worker to write to the disk- const queueInfo = newStore.toRequestQueueInfo();-- scheduleBackgroundTask(- {- action: 'update-metadata',- entityType: 'requestQueues',- entityDirectory: newStore.requestQueueDirectory,- id: queueInfo.name ?? queueInfo.id,- data: queueInfo,- writeMetadata: this.writeMetadata,- persistStorage: this.persistStorage,- },- this.logger,- );-return newStore;}async storageExists(id: string, type: 'Dataset' | 'KeyValueStore' | 'RequestQueue'): Promise<boolean> {let clients: { id: string }[];- let baseDir: string;switch (type) {case 'Dataset':clients = this.datasetClientCache;- baseDir = this.datasetsDirectory;break;case 'KeyValueStore':clients = this.keyValueStoreCache;- baseDir = this.keyValueStoresDirectory;break;case 'RequestQueue':clients = this.requestQueueCache;- baseDir = this.requestQueuesDirectory;break;default:return false;}- // Check in-memory cache by actual storage ID- if (clients.some((store) => store.id === id)) {- return true;- }-- // Check if a directory with that ID exists on disk.- // Only consider directories whose name matches the queried ID — this avoids- // false positives for alias-created directories (e.g. a directory named 'asdf'- // created via `{ alias: 'asdf' }` should not make `storageExists('asdf')` return true,- // since the actual storage ID is a UUID, not the alias string).- try {- await access(resolve(baseDir, id));-- // If the directory exists but a cached client already owns this directory- // under a different ID, this is not a match.- const cachedClients = clients as { id: string; directoryName?: string }[];- if (cachedClients.some((store) => store.directoryName === id && store.id !== id)) {- return false;- }-- return true;- } catch {- return false;- }+ // In-memory storage only knows about clients in its cache.+ return clients.some((store) => store.id === id);}async setStatusMessage(message: string, options: storage.SetStatusMessageOptions = {}): Promise<void> {@@ -286,167 +161,42 @@ export class MemoryStorage implements storage.StorageClient {}/**- * Cleans up the default storage directories before the run starts:- * - local directory containing the default dataset;- * - all records from the default key-value store in the local directory, except for the "INPUT" key;- * - local directory containing the default request queue.+ * Cleans up the default storages before the run starts. For the in-memory storage this simply+ * resets the in-memory state of the cached default dataset, key-value store and request queue.+ *+ * As with `FileSystemStorageClient`, the run's input (the `INPUT` key in the default key-value+ * store) is preserved — only the rest of the default storages is cleared.*/async purge(): Promise<void> {- // Key-value stores- const keyValueStores = await readdir(this.keyValueStoresDirectory).catch(() => []);- const keyValueStorePromises: Promise<void>[] = [];-- for (const keyValueStoreFolder of keyValueStores) {- if (keyValueStoreFolder.startsWith('__CRAWLEE_TEMPORARY') || keyValueStoreFolder.startsWith('__OLD')) {- keyValueStorePromises.push(- (await this.batchRemoveFiles(resolve(this.keyValueStoresDirectory, keyValueStoreFolder)))(),- );- } else if (keyValueStoreFolder === 'default' || keyValueStoreFolder === '__default__') {- keyValueStorePromises.push(- this.handleDefaultKeyValueStore(resolve(this.keyValueStoresDirectory, keyValueStoreFolder))(),- );- }- }-- void Promise.allSettled(keyValueStorePromises);-- // Datasets- const datasets = await readdir(this.datasetsDirectory).catch(() => []);- const datasetPromises: Promise<void>[] = [];-- for (const datasetFolder of datasets) {- if (- datasetFolder === 'default' ||- datasetFolder === '__default__' ||- datasetFolder.startsWith('__CRAWLEE_TEMPORARY')- ) {- datasetPromises.push((await this.batchRemoveFiles(resolve(this.datasetsDirectory, datasetFolder)))());- }- }-- void Promise.allSettled(datasetPromises);-- // Request queues- const requestQueues = await readdir(this.requestQueuesDirectory).catch(() => []);- const requestQueuePromises: Promise<void>[] = [];-- for (const requestQueueFolder of requestQueues) {- if (- requestQueueFolder === 'default' ||- requestQueueFolder === '__default__' ||- requestQueueFolder.startsWith('__CRAWLEE_TEMPORARY')- ) {- requestQueuePromises.push(- (await this.batchRemoveFiles(resolve(this.requestQueuesDirectory, requestQueueFolder)))(),- );- }- }+ // The run default is opened via `{ alias: '__default__' }`, which `resolveStorageKey`+ // normalizes to `cacheKey === 'default'` (with `name === undefined`) — that is the clause+ // that actually matches it. The `name === 'default'` clause additionally covers a store a user+ // explicitly opened via `{ name: 'default' }`. (`'__default__'` never reaches `cacheKey`,+ // as it is always normalized to `'default'` first, so it does not need to be checked here.)+ const isDefault = (store: { name?: string; cacheKey: string }) =>+ store.name === 'default' || store.cacheKey === 'default';++ const purgeDefaults = async <T extends { name?: string; cacheKey: string }>(+ cache: T[],+ purgeStore: (store: T) => Promise<void>,+ ) => {+ await Promise.all(cache.filter(isDefault).map(async (store) => purgeStore(store)));+ };- void Promise.allSettled(requestQueuePromises);+ await Promise.all([+ // Preserve the run input (INPUT) when purging the default key-value store, matching+ // `FileSystemStorageClient`.+ purgeDefaults(this.keyValueStoreCache, async (store) => store.purgeExceptInput()),+ purgeDefaults(this.datasetClientCache, async (store) => store.purge()),+ purgeDefaults(this.requestQueueCache, async (store) => store.purge()),+ ]);}/**- * This method should be called at the end of the process, to ensure all data is saved.+ * This method should be called at the end of the process. The in-memory storage holds no resources+ * that outlive the process (no file handles, no cross-process locks), so there is nothing to do.*/async teardown(): Promise<void> {- // Release any request locks this process still holds so that requests fetched but not yet- // handled are not stuck (until their lock expires) for the next consumer of the same on-disk- // queue. Other storage backends don't need this: the Apify platform releases a run's locks- // automatically, and the file-system storage doesn't lock at all.- await Promise.all(this.requestQueueCache.map(async (queue) => queue.releaseOwnLocks()));-- const promises = [...promiseMap.values()].map(async ({ promise }) => promise);-- await Promise.all(promises);- }-- private handleDefaultKeyValueStore(folder: string): () => Promise<void> {- const storagePathExists = pathExistsSync(folder);- const temporaryPath = resolve(folder, '../__CRAWLEE_MIGRATING_KEY_VALUE_STORE__');-- // For optimization, we want to only attempt to copy a few files from the default key-value store- const possibleInputKeys = ['INPUT', 'INPUT.json', 'INPUT.bin', 'INPUT.txt'];-- if (storagePathExists) {- // Create temporary folder to save important files in- ensureDirSync(temporaryPath);-- // Go through each file and save the ones that are important- for (const entity of possibleInputKeys) {- const originalFilePath = resolve(folder, entity);- const tempFilePath = resolve(temporaryPath, entity);-- try {- moveSync(originalFilePath, tempFilePath);- } catch {- // Ignore- }- }-- // Remove the original folder and all its content- let counter = 0;- let tempPathForOldFolder = resolve(folder, `../__OLD_DEFAULT_${counter}__`);- let done = false;-- while (!done) {- try {- moveSync(folder, tempPathForOldFolder);- done = true;- } catch {- tempPathForOldFolder = resolve(folder, `../__OLD_DEFAULT_${++counter}__`);- }- }-- // Replace the temporary folder with the original folder- moveSync(temporaryPath, folder);-- // Remove the old folder- return async () => (await this.batchRemoveFiles(tempPathForOldFolder))();- }-- return async () => Promise.resolve();- }-- private async batchRemoveFiles(folder: string, counter = 0): Promise<() => Promise<void>> {- const folderExists = pathExistsSync(folder);-- if (folderExists) {- const temporaryFolder = resolve(folder, `../__CRAWLEE_TEMPORARY_${counter}__`);-- try {- // Rename the old folder to the new one to allow background deletions- await move(folder, temporaryFolder);- } catch {- // Folder exists already, try again with an incremented counter- return this.batchRemoveFiles(folder, ++counter);- }-- return async () => {- // Read all files in the folder- const entries = await readdir(temporaryFolder);-- let processed = 0;- let promises: Promise<void>[] = [];-- for (const entry of entries) {- processed++;- promises.push(rm(resolve(temporaryFolder, entry), { force: true }));-- // Every 2000 files, delete them- if (processed % 2000 === 0) {- await Promise.allSettled(promises);- promises = [];- }- }-- // Ensure last promises are handled- await Promise.allSettled(promises);-- // Delete the folder itself- await rm(temporaryFolder, { force: true, recursive: true });- };- }-- return async () => Promise.resolve();+ // Nothing to tear down for in-memory storage.}}diff --git a/packages/memory-storage/src/resource-clients/common/base-client.ts b/packages/memory-storage/src/resource-clients/common/base-client.tsindex 3385ae3ba..72a994c2d 100644--- a/packages/memory-storage/src/resource-clients/common/base-client.ts+++ b/packages/memory-storage/src/resource-clients/common/base-client.ts@@ -1,17 +1,7 @@-import type { StorageTypes } from '../../consts.js';-export class BaseClient {id: string;constructor(id: string) {this.id = id;}-- protected throwOnNonExisting(clientType: StorageTypes): never {- throw new Error(`${clientType} with id: ${this.id} does not exist.`);- }-- protected throwOnDuplicateEntry(clientType: StorageTypes, keyName: string, value: string): never {- throw new Error(`${clientType} with ${keyName}: ${value} already exists.`);- }}diff --git a/packages/memory-storage/src/resource-clients/dataset.ts b/packages/memory-storage/src/resource-clients/dataset.tsindex 78cb8cd85..55791ced8 100644--- a/packages/memory-storage/src/resource-clients/dataset.ts+++ b/packages/memory-storage/src/resource-clients/dataset.ts@@ -1,17 +1,10 @@-/* eslint-disable import/no-duplicates */import { randomUUID } from 'node:crypto';-import { rm } from 'node:fs/promises';-import { resolve } from 'node:path';import type * as storage from '@crawlee/types';import type { Dictionary } from '@crawlee/types';import { s } from '@sapphire/shapeshift';-import { scheduleBackgroundTask } from '../background-handler/index.js';-import type { StorageImplementation } from '../fs/common.js';-import { createDatasetStorageImplementation } from '../fs/dataset/index.js';-import type { MemoryStorage } from '../index.js';-import { resolveWithinDirectory } from '../utils.js';+import type { MemoryStorageClient } from '../index.js';import { BaseClient } from './common/base-client.js';/**@@ -21,8 +14,8 @@ import { BaseClient } from './common/base-client.js';const LIST_ITEMS_LIMIT = 999_999_999_999;/**- * Number of characters of the dataset item file names.- * E.g.: 000000019.json - 9 digits+ * Number of characters of the dataset item entry names.+ * E.g.: 000000019 - 9 digits*/const LOCAL_ENTRY_NAME_DIGITS = 9;@@ -30,13 +23,12 @@ export interface DatasetClientOptions {id?: string;name?: string;/**- * The directory name to use on disk. When provided, takes precedence over `name` and `id`- * for the directory path. This allows alias-opened storages to have a directory name- * that differs from their metadata `name` (which is `undefined` for unnamed storages).+ * The key used for cache lookup. When provided, takes precedence over `name` and `id`.+ * This allows alias-opened storages to have a cache key that differs from their+ * metadata `name` (which is `undefined` for unnamed storages).*/- directoryName?: string;- baseStorageDirectory: string;- client: MemoryStorage;+ cacheKey?: string;+ client: MemoryStorageClient;}export class DatasetClient<Data extends Dictionary = Dictionary>@@ -45,24 +37,22 @@ export class DatasetClient<Data extends Dictionary = Dictionary>{name?: string;/**- * The key used for directory naming and cache lookup. For named storages, this equals- * the name. For alias (unnamed) storages, this is the alias string. Falls back to id.+ * The key used for cache lookup. For named storages, this equals the name. For alias (unnamed)+ * storages, this is the alias string. Falls back to id.*/- directoryName: string;+ cacheKey: string;createdAt = new Date();accessedAt = new Date();modifiedAt = new Date();itemCount = 0;- datasetDirectory: string;- private readonly datasetEntries = new Map<string, StorageImplementation<Data>>();- private readonly client: MemoryStorage;+ private readonly datasetEntries = new Map<string, Data>();+ private readonly client: MemoryStorageClient;constructor(options: DatasetClientOptions) {super(options.id ?? randomUUID());this.name = options.name;- this.directoryName = options.directoryName ?? this.name ?? this.id;- this.datasetDirectory = resolveWithinDirectory(options.baseStorageDirectory, this.directoryName);+ this.cacheKey = options.cacheKey ?? this.name ?? this.id;this.client = options.client;}@@ -78,8 +68,6 @@ export class DatasetClient<Data extends Dictionary = Dictionary>const [oldClient] = this.client.datasetClientCache.splice(storeIndex, 1);oldClient.itemCount = 0;oldClient.datasetEntries.clear();-- await rm(oldClient.datasetDirectory, { recursive: true, force: true });}}@@ -87,17 +75,6 @@ export class DatasetClient<Data extends Dictionary = Dictionary>this.itemCount = 0;this.datasetEntries.clear();- // Remove item files from disk but keep the directory- if (this.client.persistStorage) {- const { readdir } = await import('node:fs/promises');- const entries = await readdir(this.datasetDirectory).catch(() => []);- for (const entry of entries) {- if (entry !== '__metadata__.json') {- await rm(resolve(this.datasetDirectory, entry), { force: true });- }- }- }-this.updateTimestamps(true);}@@ -129,7 +106,7 @@ export class DatasetClient<Data extends Dictionary = Dictionary>for (let idx = start; idx < end; idx++) {const entryNumber = this.generateLocalEntryName(idx);- items.push(await this.datasetEntries.get(entryNumber)!.get());+ items.push(this.datasetEntries.get(entryNumber)!);}this.updateTimestamps(false);@@ -147,15 +124,7 @@ export class DatasetClient<Data extends Dictionary = Dictionary>async pushData(items: Data[]): Promise<void> {for (const entry of items) {const idx = this.generateLocalEntryName(++this.itemCount);- const storageEntry = createDatasetStorageImplementation({- entityId: idx,- persistStorage: this.client.persistStorage,- storeDirectory: this.datasetDirectory,- });-- await storageEntry.update(entry);-- this.datasetEntries.set(idx, storageEntry);+ this.datasetEntries.set(idx, entry);}this.updateTimestamps(true);@@ -188,19 +157,5 @@ export class DatasetClient<Data extends Dictionary = Dictionary>if (hasBeenModified) {this.modifiedAt = new Date();}-- const data = this.toDatasetInfo();- scheduleBackgroundTask(- {- action: 'update-metadata',- data,- entityType: 'datasets',- entityDirectory: this.datasetDirectory,- id: this.name ?? this.id,- writeMetadata: this.client.writeMetadata,- persistStorage: this.client.persistStorage,- },- this.client.logger,- );}}diff --git a/packages/memory-storage/src/resource-clients/key-value-store.ts b/packages/memory-storage/src/resource-clients/key-value-store.tsindex f11138c98..14e8ef273 100644--- a/packages/memory-storage/src/resource-clients/key-value-store.ts+++ b/packages/memory-storage/src/resource-clients/key-value-store.ts@@ -1,31 +1,32 @@import { randomUUID } from 'node:crypto';-import { rm } from 'node:fs/promises';import type * as storage from '@crawlee/types';import { s } from '@sapphire/shapeshift';-import { scheduleBackgroundTask } from '../background-handler/index.js';import { maybeParseBody } from '../body-parser.js';-import type { StorageImplementation } from '../fs/common.js';-import { createKeyValueStorageImplementation } from '../fs/key-value-store/index.js';-import type { MemoryStorage } from '../index.js';-import { isBuffer, isStream, resolveWithinDirectory } from '../utils.js';+import type { MemoryStorageClient } from '../index.js';+import { isBuffer, isStream } from '../utils.js';import { BaseClient } from './common/base-client.js';import mime from 'mime-types';const DEFAULT_LOCAL_FILE_EXTENSION = 'bin';+/**+ * Key under which a run's input is stored in the default key-value store. Matches Crawlee's default+ * `inputKey` (`CRAWLEE_INPUT_KEY`) and the `INPUT` files `FileSystemStorageClient` preserves on purge.+ */+const KEY_VALUE_STORE_INPUT_KEY = 'INPUT';+export interface KeyValueStoreClientOptions {name?: string;id?: string;/**- * The directory name to use on disk. When provided, takes precedence over `name` and `id`- * for the directory path. This allows alias-opened storages to have a directory name- * that differs from their metadata `name` (which is `undefined` for unnamed storages).+ * The key used for cache lookup. When provided, takes precedence over `name` and `id`.+ * This allows alias-opened storages to have a cache key that differs from their+ * metadata `name` (which is `undefined` for unnamed storages).*/- directoryName?: string;- baseStorageDirectory: string;- client: MemoryStorage;+ cacheKey?: string;+ client: MemoryStorageClient;}export interface InternalKeyRecord {@@ -33,29 +34,26 @@ export interface InternalKeyRecord {value: Buffer | string;contentType?: string;extension: string;- filePath?: string;}export class KeyValueStoreClient extends BaseClient implements storage.KeyValueStoreClient {name?: string;/**- * The key used for directory naming and cache lookup. For named storages, this equals- * the name. For alias (unnamed) storages, this is the alias string. Falls back to id.+ * The key used for cache lookup. For named storages, this equals the name. For alias (unnamed)+ * storages, this is the alias string. Falls back to id.*/- directoryName: string;+ cacheKey: string;createdAt = new Date();accessedAt = new Date();modifiedAt = new Date();- keyValueStoreDirectory: string;- private readonly keyValueEntries = new Map<string, StorageImplementation<InternalKeyRecord>>();- private readonly client: MemoryStorage;+ private readonly keyValueEntries = new Map<string, InternalKeyRecord>();+ private readonly client: MemoryStorageClient;constructor(options: KeyValueStoreClientOptions) {super(options.id ?? randomUUID());this.name = options.name;- this.directoryName = options.directoryName ?? this.name ?? this.id;- this.keyValueStoreDirectory = resolveWithinDirectory(options.baseStorageDirectory, this.directoryName);+ this.cacheKey = options.cacheKey ?? this.name ?? this.id;this.client = options.client;}@@ -70,17 +68,25 @@ export class KeyValueStoreClient extends BaseClient implements storage.KeyValueSif (storeIndex !== -1) {const [oldClient] = this.client.keyValueStoreCache.splice(storeIndex, 1);oldClient.keyValueEntries.clear();-- await rm(oldClient.keyValueStoreDirectory, { recursive: true, force: true });}}async purge(): Promise<void> {- // Delete all entries- const entriesToDelete = [...this.keyValueEntries.entries()];- for (const [key, entry] of entriesToDelete) {- this.keyValueEntries.delete(key);- await entry.delete();+ this.keyValueEntries.clear();+ this.updateTimestamps(true);+ }++ /**+ * Purges every record except the run's input. Used by {@link MemoryStorageClient.purge} for the+ * default key-value store, mirroring `FileSystemStorageClient`, which preserves `INPUT` (and its+ * extension variants) when purging the default store. The in-memory key has no extension, so we+ * preserve the bare `INPUT` key only.+ */+ async purgeExceptInput(): Promise<void> {+ for (const key of this.keyValueEntries.keys()) {+ if (key !== KEY_VALUE_STORE_INPUT_KEY) {+ this.keyValueEntries.delete(key);+ }}this.updateTimestamps(true);@@ -97,9 +103,7 @@ export class KeyValueStoreClient extends BaseClient implements storage.KeyValueSconst items: storage.KeyValueStoreItemData[] = [];- for (const storageEntry of this.keyValueEntries.values()) {- const record = await storageEntry.get();-+ for (const record of this.keyValueEntries.values()) {const size = Buffer.byteLength(record.value);items.push({key: record.key,@@ -133,17 +137,14 @@ export class KeyValueStoreClient extends BaseClient implements storage.KeyValueS}/**- * Generates a public file:// URL for accessing a specific record in the key-value store.- *- * Returns `undefined` if the record does not exist or has no associated file path (i.e., it is not stored as a file).+ * In-memory records are not file-backed, so there is no public file URL to return.+ * Always resolves to `undefined`.* @param key The key of the record to generate the public URL for.*/async getPublicUrl(key: string): Promise<string | undefined> {s.string().parse(key);- const storageEntry = await this.keyValueEntries.get(key)?.get();-- return storageEntry?.filePath;+ return undefined;}/**@@ -161,14 +162,12 @@ export class KeyValueStoreClient extends BaseClient implements storage.KeyValueSasync getValue(key: string): Promise<storage.KeyValueStoreRecord | undefined> {s.string().parse(key);- const storageEntry = this.keyValueEntries.get(key);+ const entry = this.keyValueEntries.get(key);- if (!storageEntry) {+ if (!entry) {return undefined;}- const entry = await storageEntry.get();-const record: storage.KeyValueStoreRecord = {key: entry.key,value: entry.value,@@ -240,16 +239,7 @@ export class KeyValueStoreClient extends BaseClient implements storage.KeyValueScontentType,} satisfies InternalKeyRecord;- const entry = createKeyValueStorageImplementation({- persistStorage: this.client.persistStorage,- storeDirectory: this.keyValueStoreDirectory,- writeMetadata: this.client.writeMetadata,- logger: this.client.logger,- });-- await entry.update(_record);-- this.keyValueEntries.set(key, entry);+ this.keyValueEntries.set(key, _record);this.updateTimestamps(true);}@@ -257,12 +247,9 @@ export class KeyValueStoreClient extends BaseClient implements storage.KeyValueSasync deleteValue(key: string): Promise<void> {s.string().parse(key);- const entry = this.keyValueEntries.get(key);-- if (entry) {+ if (this.keyValueEntries.has(key)) {this.keyValueEntries.delete(key);this.updateTimestamps(true);- await entry.delete();}}@@ -283,19 +270,5 @@ export class KeyValueStoreClient extends BaseClient implements storage.KeyValueSif (hasBeenModified) {this.modifiedAt = new Date();}-- const data = this.toKeyValueStoreInfo();- scheduleBackgroundTask(- {- action: 'update-metadata',- data,- entityType: 'keyValueStores',- entityDirectory: this.keyValueStoreDirectory,- id: this.name ?? this.id,- writeMetadata: this.client.writeMetadata,- persistStorage: this.client.persistStorage,- },- this.client.logger,- );}}diff --git a/packages/memory-storage/src/resource-clients/request-queue.ts b/packages/memory-storage/src/resource-clients/request-queue.tsindex cb5d40b37..5167328fa 100644--- a/packages/memory-storage/src/resource-clients/request-queue.ts+++ b/packages/memory-storage/src/resource-clients/request-queue.ts@@ -1,16 +1,10 @@import { randomUUID } from 'node:crypto';-import { rm } from 'node:fs/promises';import type * as storage from '@crawlee/types';import { AsyncQueue } from '@sapphire/async-queue';import { s } from '@sapphire/shapeshift';--import { scheduleBackgroundTask } from '../background-handler/index.js';-import { createRequestQueueStorageImplementation } from '../fs/request-queue/index.js';-import type { RequestQueueFileSystemEntry } from '../fs/request-queue/fs.js';-import type { RequestQueueMemoryEntry } from '../fs/request-queue/memory.js';-import type { MemoryStorage } from '../index.js';-import { purgeNullsFromObject, resolveWithinDirectory, uniqueKeyToRequestId } from '../utils.js';+import type { MemoryStorageClient } from '../index.js';+import { purgeNullsFromObject, uniqueKeyToRequestId } from '../utils.js';import { BaseClient } from './common/base-client.js';const requestShape = s@@ -36,13 +30,12 @@ export interface RequestQueueClientOptions {name?: string;id?: string;/**- * The directory name to use on disk. When provided, takes precedence over `name` and `id`- * for the directory path. This allows alias-opened storages to have a directory name- * that differs from their metadata `name` (which is `undefined` for unnamed storages).+ * The key used for cache lookup. When provided, takes precedence over `name` and `id`.+ * This allows alias-opened storages to have a cache key that differs from their+ * metadata `name` (which is `undefined` for unnamed storages).*/- directoryName?: string;- baseStorageDirectory: string;- client: MemoryStorage;+ cacheKey?: string;+ client: MemoryStorageClient;}export interface InternalRequest {@@ -55,41 +48,23 @@ export interface InternalRequest {json: string;}-/**- * Default time (in seconds) for which a request fetched via {@link RequestQueueClient.fetchNextRequest}- * stays locked (in progress) before it becomes available again. Aligns with the historical request queue- * locking default. A consumer (e.g. a crawler) can raise this per queue via- * {@link RequestQueueClient.setExpectedRequestProcessingTimeSecs}.- */-const DEFAULT_REQUEST_LOCK_SECS = 3 * 60;--/**- * A request is "locked" (in progress) when its `orderNo` is pushed beyond the current time. The sign of- * `orderNo` is preserved so the original forefront / normal ordering is restored once the lock expires- * or the request is reclaimed.- */-function isRequestLocked(orderNo: number | null, now: number): boolean {- return orderNo !== null && Math.abs(orderNo) > now;-}-export class RequestQueueClient extends BaseClient implements storage.RequestQueueClient {name?: string;/**- * The key used for directory naming and cache lookup. For named storages, this equals- * the name. For alias (unnamed) storages, this is the alias string. Falls back to id.+ * The key used for cache lookup. For named storages, this equals the name. For alias (unnamed)+ * storages, this is the alias string. Falls back to id.*/- directoryName: string;+ cacheKey: string;createdAt = new Date();accessedAt = new Date();modifiedAt = new Date();handledRequestCount = 0;pendingRequestCount = 0;- requestQueueDirectory: string;/*** Serializes every operation that reads-then-writes this client's shared queue state — the* `requests` map, the `forefrontRequestIds` array, the `inProgressRequestIds` set and the request- * counts. Those mutations span `await` points (disk/storage I/O), so without this lock a concurrent- * operation could interleave and corrupt them (e.g. a head scan pruning `forefrontRequestIds` while+ * counts. Those mutations span `await` points, so without this mutex a concurrent operation could+ * interleave and corrupt them (e.g. a head scan pruning `forefrontRequestIds` while* `addBatchOfRequests` pushes to it). Held by every mutating method as well as by `isEmpty`/* `isFinished`, whose head scan also prunes `forefrontRequestIds`.*/@@ -97,52 +72,51 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQueprivate forefrontRequestIds: string[] = [];/**- * IDs of requests this client has locked (fetched but not yet handled or reclaimed). Used by- * {@link releaseOwnLocks} to free our own in-progress requests on process termination, so that- * a crashed/migrated consumer does not block its requests for the full lock duration.+ * IDs of requests currently fetched but not yet handled or reclaimed. A request in this set is+ * "in progress" and will not be handed out again by {@link fetchNextRequest}.+ *+ * Unlike the file-system / platform clients, the in-memory queue lives entirely within a single+ * process and is never shared with another consumer, so there is no need for an expiring,+ * cross-process-visible lock — tracking in-progress requests in this set is enough.*/private readonly inProgressRequestIds = new Set<string>();- private readonly requests = new Map<string, RequestQueueFileSystemEntry | RequestQueueMemoryEntry>();- private readonly client: MemoryStorage;-- /**- * How long (in seconds) a request fetched from this client stays locked (in progress). Defaults to- * {@link DEFAULT_REQUEST_LOCK_SECS} and is overridable via {@link setExpectedRequestProcessingTimeSecs}.- */- private lockSecs = DEFAULT_REQUEST_LOCK_SECS;+ private readonly requests = new Map<string, InternalRequest>();+ private readonly client: MemoryStorageClient;constructor(options: RequestQueueClientOptions) {super(options.id ?? randomUUID());this.name = options.name;- this.directoryName = options.directoryName ?? this.name ?? this.id;- this.requestQueueDirectory = resolveWithinDirectory(options.baseStorageDirectory, this.directoryName);+ this.cacheKey = options.cacheKey ?? this.name ?? this.id;this.client = options.client;}- /**- * Applies how long {@link fetchNextRequest} locks a request before it becomes available again. The- * caller (the `RequestQueue` frontend) owns the policy of what this value should be — this method- * just applies it.- */- setExpectedRequestProcessingTimeSecs(secs: number): void {- this.lockSecs = secs;- }-async getMetadata(): Promise<storage.RequestQueueInfo> {this.updateTimestamps(false);return this.toRequestQueueInfo();}async drop(): Promise<void> {- const storeIndex = this.client.requestQueueCache.findIndex((queue) => queue.id === this.id);-- if (storeIndex !== -1) {- const [oldClient] = this.client.requestQueueCache.splice(storeIndex, 1);- oldClient.pendingRequestCount = 0;- oldClient.requests.clear();+ // Serialize against other mutators (and the head scans in `isEmpty`/`isFinished`) so a concurrent+ // operation cannot observe half-cleared state — e.g. a forefront id whose request has already been+ // removed, which `listPendingHead` would then dereference as `undefined`.+ await this.queueStateMutex.wait();- await rm(oldClient.requestQueueDirectory, { recursive: true, force: true });+ try {+ const storeIndex = this.client.requestQueueCache.findIndex((queue) => queue.id === this.id);++ if (storeIndex !== -1) {+ const [oldClient] = this.client.requestQueueCache.splice(storeIndex, 1);+ oldClient.pendingRequestCount = 0;+ // Clear all in-memory state, consistent with `purge`. Clearing `requests` alone would+ // leave dangling ids in `forefrontRequestIds`/`inProgressRequestIds`, which a later head+ // scan would resolve to a missing request and dereference.+ oldClient.requests.clear();+ oldClient.forefrontRequestIds = [];+ oldClient.inProgressRequestIds.clear();+ }+ } finally {+ this.queueStateMutex.shift();}}@@ -159,21 +133,6 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQuethis.handledRequestCount = 0;this.pendingRequestCount = 0;- // Reset the lock duration back to the default so a value raised via- // `setExpectedRequestProcessingTimeSecs` in an earlier run does not leak into a later one- this.lockSecs = DEFAULT_REQUEST_LOCK_SECS;-- // Remove request files from disk but keep the directory- if (this.client.persistStorage) {- const { readdir } = await import('node:fs/promises');- const entries = await readdir(this.requestQueueDirectory).catch(() => []);- for (const entry of entries) {- if (entry !== '__metadata__.json') {- await rm(resolve(this.requestQueueDirectory, entry), { force: true });- }- }- }-this.updateTimestamps(true);} finally {this.queueStateMutex.shift();@@ -192,32 +151,25 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQue/*** Scans the queue and returns the pending head — requests that are neither handled nor currently- * locked (in progress) — ordered by `orderNo`, deduplicated.- *- * When `detectLockedRequests` is set, the result also carries a `hasLockedRequests` flag telling- * whether any unhandled-but-locked request was skipped along the way. This mirrors the Apify- * platform shared client's `queueHasLockedRequests` signal: it lets {@link isFinished} distinguish- * "no work left at all" from "work remains, but it is currently locked by some consumer (possibly- * another process)". Without it, a consumer would consider the queue finished and let the crawler- * shut down while another consumer still holds the last requests.+ * in progress — ordered by `orderNo`, deduplicated.*- * Computing the flag is expensive: because a lock may sit anywhere in the queue, it forces a scan- * of every pending entry even when only `limit` items are wanted. Callers that only need the head- * (e.g. {@link fetchNextRequest}, {@link isEmpty}) leave it off so the scan can stop as soon as the- * page is filled, keeping those calls O(head) instead of O(N).+ * When `detectInProgressRequests` is set, the result also carries an `hasInProgressRequests` flag+ * telling whether any unhandled-but-in-progress request was skipped along the way. It lets+ * {@link isFinished} distinguish "no work left at all" from "work remains, but it is currently being+ * processed". Without it, a consumer with concurrency could consider the queue finished and shut the+ * crawler down while it is still handling the last requests.*- * Lock state lives in the persisted `orderNo` (see {@link isRequestLocked}), so that processes- * sharing the same on-disk queue observe each other's locks. We therefore re-read entries from- * storage to obtain fresh lock state, except for entries we can cheaply rule out as permanently- * handled via their cached `orderNo === null`.+ * Computing the flag is expensive: because an in-progress request may sit anywhere in the queue, it+ * forces a scan of every pending entry even when only `limit` items are wanted. Callers that only+ * need the head (e.g. {@link fetchNextRequest}, {@link isEmpty}) leave it off so the scan can stop as+ * soon as the page is filled, keeping those calls O(head) instead of O(N).*/private async listPendingHead(limit: number,- detectLockedRequests = false,- ): Promise<{ items: InternalRequest[]; hasLockedRequests?: boolean }> {- const now = Date.now();+ detectInProgressRequests = false,+ ): Promise<{ items: InternalRequest[]; hasInProgressRequests?: boolean }> {const items: InternalRequest[] = [];- let hasLockedRequests = false;+ let hasInProgressRequests = false;// Tracks processed request IDs to avoid duplicates (request in both `forefrontRequestIds` and `requests`).const seenRequestIds = new Set<string>();@@ -225,9 +177,9 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQueconst handledForefrontIds = new Set<string>();for (const requestId of this.requestKeyIterator()) {- // Once the requested page is filled we can stop — unless the caller asked us to detect locked- // requests and we have not yet seen one, in which case we must keep scanning to find them.- if (items.length >= limit && (!detectLockedRequests || hasLockedRequests)) {+ // Once the requested page is filled we can stop — unless the caller asked us to detect+ // in-progress requests and we have not yet seen one, in which case we must keep scanning.+ if (items.length >= limit && (!detectInProgressRequests || hasInProgressRequests)) {break;}@@ -237,22 +189,9 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQueseenRequestIds.add(requestId);- const storageEntry = this.requests.get(requestId)!;+ const request = this.requests.get(requestId)!;- // Cheap rejection of permanently-handled requests using the cached `orderNo` (handled is a- // terminal state, so the cached value can be trusted without re-reading from storage).- if (storageEntry.orderNo === null) {- if (this.forefrontRequestIds.includes(requestId)) {- handledForefrontIds.add(requestId);- }- continue;- }-- // Re-read from storage to get fresh lock state — another process may have locked (or handled)- // this request since we last cached it.- const request = await storageEntry.get(true);-- // Handled in the meantime.+ // Permanently-handled requests (`orderNo === null`) are in a terminal state and can be skipped.if (request.orderNo === null) {if (this.forefrontRequestIds.includes(requestId)) {handledForefrontIds.add(requestId);@@ -260,10 +199,10 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQuecontinue;}- // Locked (in progress) by us or another process — skip until the lock expires, but remember- // that the queue is not truly empty.- if (isRequestLocked(request.orderNo, now)) {- hasLockedRequests = true;+ // In progress (fetched but not yet handled or reclaimed) — skip it, but remember that the+ // queue is not truly empty.+ if (this.inProgressRequestIds.has(requestId)) {+ hasInProgressRequests = true;continue;}@@ -276,7 +215,7 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQuereturn {items: items.sort((a, b) => a.orderNo! - b.orderNo!),- hasLockedRequests: detectLockedRequests ? hasLockedRequests : undefined,+ hasInProgressRequests: detectInProgressRequests ? hasInProgressRequests : undefined,};}@@ -294,15 +233,8 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQuereturn null;}- // Lock the request by pushing its `orderNo` beyond the lock expiry, preserving the sign so- // its original (forefront / normal) position is restored once the lock expires. The lock is- // persisted so other processes sharing this queue will not fetch the same request.- const lockExpiresAt = Date.now() + this.lockSecs * 1000;- head.orderNo = lockExpiresAt * (head.orderNo! > 0 ? 1 : -1);- await this.requests.get(head.id)!.update(head);-- // Remember that this client owns the lock, so we can release it on process termination- // (see `releaseOwnLocks`) instead of leaving the request stuck until the lock expires.+ // Mark the request as in progress so it is not handed out again until it is handled or+ // reclaimed. The request keeps its `orderNo` (and thus its forefront / normal ordering).this.inProgressRequestIds.add(head.id);return this._jsonToRequest(head.json) ?? null;@@ -332,11 +264,9 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQuefor (const model of requests) {const requestModel = this._createInternalRequest(model, options.forefront);- const existingRequestWithIdEntry = this.requests.get(requestModel.id);-- if (existingRequestWithIdEntry) {- const existingRequestWithId = await existingRequestWithIdEntry.get();+ const existingRequestWithId = this.requests.get(requestModel.id);+ if (existingRequestWithId) {result.processedRequests.push({requestId: existingRequestWithId.id,uniqueKey: existingRequestWithId.uniqueKey,@@ -347,15 +277,7 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQuecontinue;}- const newEntry = createRequestQueueStorageImplementation({- persistStorage: this.client.persistStorage,- requestId: requestModel.id,- storeDirectory: this.requestQueueDirectory,- });-- await newEntry.update(requestModel);-- this.requests.set(requestModel.id, newEntry);+ this.requests.set(requestModel.id, requestModel);if (requestModel.orderNo) {this.pendingRequestCount += 1;@@ -389,7 +311,7 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQues.string().parse(uniqueKey);this.updateTimestamps(false);const id = uniqueKeyToRequestId(uniqueKey);- const json = (await this.requests.get(id)?.get())?.json;+ const json = this.requests.get(id)?.json;return this._jsonToRequest(json);}@@ -405,13 +327,11 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQuetry {const id = uniqueKeyToRequestId(request.uniqueKey);- const existingEntry = this.requests.get(id);- const existingRequest = await existingEntry?.get();+ const existingRequest = this.requests.get(id);// The request must exist to be marked as handled. We intentionally do NOT require it to still- // be locked: a consumer whose processing outlived the lock (slow handler, GC/event-loop pause,- // a low `setExpectedRequestProcessingTimeSecs`) must still be able to mark its request handled,- // otherwise the request would be handed out again forever and the queue would never finish.+ // be in progress: marking an already-released request handled must still succeed, otherwise+ // the request could be handed out again and the queue would never finish.if (!existingRequest) {return null;}@@ -422,13 +342,7 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQueconst handledAt = request.handledAt ?? new Date().toISOString();const requestModel = this._createInternalRequest({ ...request, handledAt }, false);- const newEntry = createRequestQueueStorageImplementation({- persistStorage: this.client.persistStorage,- requestId: id,- storeDirectory: this.requestQueueDirectory,- });- await newEntry.update(requestModel);- this.requests.set(id, newEntry);+ this.requests.set(id, requestModel);// The request is no longer in progress for this client.this.inProgressRequestIds.delete(id);@@ -466,28 +380,21 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQuetry {const id = uniqueKeyToRequestId(request.uniqueKey);- const existingEntry = this.requests.get(id);- const existingRequest = await existingEntry?.get();+ const existingRequest = this.requests.get(id);// The request must exist and not already be handled to be reclaimed. As with- // `markRequestAsHandled`, we do NOT require it to still be locked — a consumer that failed- // after its lock expired must still be able to return the request to the queue (e.g. to honor- // a `forefront` reorder), rather than have the reclaim silently dropped.+ // `markRequestAsHandled`, we do NOT require it to still be in progress — returning an+ // already-released request to the queue (e.g. to honor a `forefront` reorder) must still+ // work, rather than have the reclaim silently dropped.if (!existingRequest || existingRequest.orderNo === null) {return null;}- // Reclaiming resets the `orderNo` to a fresh timestamp, releasing the lock and restoring the- // request to the queue (at the front if `forefront`).+ // Reclaiming resets the `orderNo` to a fresh timestamp, restoring the request to the queue+ // (at the front if `forefront`).const requestModel = this._createInternalRequest(request, options.forefront);- const newEntry = createRequestQueueStorageImplementation({- persistStorage: this.client.persistStorage,- requestId: id,- storeDirectory: this.requestQueueDirectory,- });- await newEntry.update(requestModel);- this.requests.set(id, newEntry);+ this.requests.set(id, requestModel);// The request is no longer in progress for this client.this.inProgressRequestIds.delete(id);@@ -512,10 +419,10 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQuethis.updateTimestamps(false);// "Empty" means there is nothing left to fetch right now — i.e. the next `fetchNextRequest`- // would return `null`. Requests that are currently locked (in progress) are intentionally NOT- // counted here: they are not fetchable, so the queue is empty from a consumer's point of view.- // Whether those in-progress requests mean crawling is not yet done is a separate question,- // answered by `isFinished`.+ // would return `null`. Requests that are currently in progress are intentionally NOT counted+ // here: they are not fetchable, so the queue is empty from a consumer's point of view. Whether+ // those in-progress requests mean crawling is not yet done is a separate question, answered by+ // `isFinished`.//// `listPendingHead` prunes `forefrontRequestIds` as it scans, so we must hold the queue-state mutex to avoid// racing a concurrent mutator (e.g. `addBatchOfRequests`) at its `await` points.@@ -532,65 +439,20 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQueasync isFinished(): Promise<boolean> {this.updateTimestamps(false);- // The queue is finished only when there is nothing left to fetch AND nothing currently locked- // (in progress) by any consumer. Counting locked requests is what allows a crawler to keep- // waiting while another consumer (possibly another process sharing this on-disk queue) still- // holds the last requests, instead of finishing prematurely. This mirrors the Apify platform- // shared client's `queueHasLockedRequests` signal.+ // The queue is finished only when there is nothing left to fetch AND nothing currently in+ // progress. Counting in-progress requests is what allows a crawler with concurrency to keep+ // waiting while it still holds the last requests, instead of finishing prematurely.//- // Detecting locked requests requires a full scan, hence the `detectLockedRequests` flag — unlike- // `fetchNextRequest`/`isEmpty`, which only need the head and can stop early.+ // Detecting in-progress requests requires a full scan, hence the `detectInProgressRequests`+ // flag — unlike `fetchNextRequest`/`isEmpty`, which only need the head and can stop early.//// `listPendingHead` prunes `forefrontRequestIds` as it scans, so we must hold the queue-state mutex to avoid// racing a concurrent mutator (e.g. `addBatchOfRequests`) at its `await` points.await this.queueStateMutex.wait();try {- const { items, hasLockedRequests } = await this.listPendingHead(1, true);- return items.length === 0 && !hasLockedRequests;- } finally {- this.queueStateMutex.shift();- }- }-- /**- * Release the locks of all requests this client currently has in progress, returning them to the- * queue so they can be fetched again immediately.- *- * On the Apify platform, a run's locks are released automatically when it migrates or aborts, and- * the file-system storage does not lock at all — so only this in-memory client, which persists- * fake locks via `orderNo`, needs to clean up after itself. `MemoryStorage.teardown()` calls this- * for every cached queue at the end of the process so that a fetched-but-unhandled request is not- * stuck (waiting for its lock to expire) for the next consumer of the same on-disk queue.- */- async releaseOwnLocks(): Promise<void> {- if (this.inProgressRequestIds.size === 0) {- return;- }-- await this.queueStateMutex.wait();-- try {- const now = Date.now();-- for (const id of this.inProgressRequestIds) {- const entry = this.requests.get(id);- const request = await entry?.get(true);-- // Skip requests that were handled or whose lock already expired/changed — we only undo- // locks we still hold.- if (!request || !isRequestLocked(request.orderNo, now)) {- continue;- }-- // Reset the lock to a fresh timestamp, preserving the sign so the request keeps its- // original (forefront / normal) ordering once unlocked.- request.orderNo = now * (request.orderNo! > 0 ? 1 : -1);- await entry!.update(request);- }-- this.inProgressRequestIds.clear();- this.updateTimestamps(true);+ const { items, hasInProgressRequests } = await this.listPendingHead(1, true);+ return items.length === 0 && !hasInProgressRequests;} finally {this.queueStateMutex.shift();}@@ -618,24 +480,6 @@ export class RequestQueueClient extends BaseClient implements storage.RequestQueif (hasBeenModified) {this.modifiedAt = new Date();}-- const data = {- ...this.toRequestQueueInfo(),- forefrontRequestIds: this.forefrontRequestIds,- };-- scheduleBackgroundTask(- {- action: 'update-metadata',- data,- entityType: 'requestQueues',- entityDirectory: this.requestQueueDirectory,- id: this.name ?? this.id,- writeMetadata: this.client.writeMetadata,- persistStorage: this.client.persistStorage,- },- this.client.logger,- );}private _jsonToRequest<T>(requestJson?: string): T | undefined {diff --git a/packages/memory-storage/src/utils.ts b/packages/memory-storage/src/utils.tsindex eb96a77ba..0991b0af7 100644--- a/packages/memory-storage/src/utils.ts+++ b/packages/memory-storage/src/utils.ts@@ -1,7 +1,6 @@import { createHash } from 'node:crypto';import { resolve, sep } from 'node:path';-import type * as storage from '@crawlee/types';import { s } from '@sapphire/shapeshift';import { REQUEST_ID_LENGTH } from './consts.js';@@ -68,22 +67,3 @@ export function isStream(value: any): boolean {['on', 'pipe'].every((key) => key in value && typeof value[key] === 'function'));}--export type BackgroundHandlerReceivedMessage = BackgroundHandlerUpdateMetadataMessage;--export type BackgroundHandlerUpdateMetadataMessage =- | MetadataUpdate<'datasets', storage.DatasetInfo>- | MetadataUpdate<'keyValueStores', storage.KeyValueStoreInfo>- | MetadataUpdate<'requestQueues', storage.RequestQueueInfo>;--type EntityType = 'datasets' | 'keyValueStores' | 'requestQueues';--interface MetadataUpdate<Type extends EntityType, DataType> {- entityType: Type;- id: string;- action: 'update-metadata';- entityDirectory: string;- data: DataType;- writeMetadata: boolean;- persistStorage: boolean;-}diff --git a/packages/memory-storage/test/async-iteration.test.ts b/packages/memory-storage/test/async-iteration.test.tsindex 8fc6f96f6..3105c1b46 100644--- a/packages/memory-storage/test/async-iteration.test.ts+++ b/packages/memory-storage/test/async-iteration.test.ts@@ -1,19 +1,8 @@-import { rm } from 'node:fs/promises';-import path from 'node:path';--import { MemoryStorage } from '@crawlee/memory-storage';+import { MemoryStorageClient } from '@crawlee/memory-storage';import type { DatasetClient, KeyValueStoreClient } from '@crawlee/types';describe('Async iteration support', () => {- const localDataDirectory = path.resolve(__dirname, './tmp/async-iteration');- const storage = new MemoryStorage({- localDataDirectory,- persistStorage: false,- });-- afterAll(async () => {- await rm(localDataDirectory, { force: true, recursive: true });- });+ const storage = new MemoryStorageClient();describe('Dataset.getData', () => {const elements = Array.from({ length: 25 }, (_, i) => ({ index: i }));diff --git a/packages/memory-storage/test/key-value-store/purge.test.ts b/packages/memory-storage/test/key-value-store/purge.test.tsnew file mode 100644index 000000000..e64a5371a--- /dev/null+++ b/packages/memory-storage/test/key-value-store/purge.test.ts@@ -0,0 +1,38 @@+import { MemoryStorageClient } from '@crawlee/memory-storage';+import type { KeyValueStoreClient } from '@crawlee/types';++describe('MemoryStorageClient.purge preserves the default key-value store input', () => {+ test('purging keeps INPUT in the default store but removes everything else', async () => {+ const storage = new MemoryStorageClient();+ const store: KeyValueStoreClient = await storage.createKeyValueStoreClient({ name: 'default' });++ await store.setValue({ key: 'INPUT', value: { hello: 'world' } });+ await store.setValue({ key: 'some-other-key', value: { foo: 'bar' } });++ await storage.purge();++ // INPUT must survive the purge (parity with FileSystemStorageClient)...+ const input = await store.getValue('INPUT');+ expect(input?.value).toEqual({ hello: 'world' });++ // ...while every other record is removed.+ expect(await store.getValue('some-other-key')).toBeUndefined();+ const keys = await store.listKeys();+ expect(keys.map((item) => item.key)).toEqual(['INPUT']);+ });++ test('purging a non-default store removes INPUT as well', async () => {+ const storage = new MemoryStorageClient();+ const store: KeyValueStoreClient = await storage.createKeyValueStoreClient({ name: 'not-default' });++ await store.setValue({ key: 'INPUT', value: { hello: 'world' } });++ // `purge` on the storage client only touches default storages, so a named store keeps its data.+ await storage.purge();+ expect((await store.getValue('INPUT'))?.value).toEqual({ hello: 'world' });++ // Purging the store directly clears everything, including INPUT.+ await store.purge();+ expect(await store.getValue('INPUT')).toBeUndefined();+ });+});diff --git a/packages/memory-storage/test/key-value-store/stream.test.ts b/packages/memory-storage/test/key-value-store/stream.test.tsindex bfbd05cd3..327d6cbd9 100644--- a/packages/memory-storage/test/key-value-store/stream.test.ts+++ b/packages/memory-storage/test/key-value-store/stream.test.ts@@ -1,11 +1,9 @@import { Readable } from 'node:stream';-import { MemoryStorage } from '@crawlee/memory-storage';+import { MemoryStorageClient } from '@crawlee/memory-storage';describe('KeyValueStore should drain streams when setting records', () => {- const storage = new MemoryStorage({- persistStorage: false,- });+ const storage = new MemoryStorageClient();const fsStream = Readable.from([Buffer.from('hello'), Buffer.from('world')]);diff --git a/packages/memory-storage/test/no-crash-on-big-buffers.test.ts b/packages/memory-storage/test/no-crash-on-big-buffers.test.tsindex 136960b14..d7ee48435 100644--- a/packages/memory-storage/test/no-crash-on-big-buffers.test.ts+++ b/packages/memory-storage/test/no-crash-on-big-buffers.test.ts@@ -1,18 +1,11 @@// https://github.com/apify/crawlee/issues/1732// https://github.com/apify/crawlee/issues/1710-import { rm } from 'node:fs/promises';-import { resolve } from 'node:path';--import { MemoryStorage } from '@crawlee/memory-storage';+import { MemoryStorageClient } from '@crawlee/memory-storage';import type { KeyValueStoreClient } from '@crawlee/types';-describe('MemoryStorage should not crash when saving a big buffer', () => {- const tmpLocation = resolve(import.meta.dirname, './tmp/no-buffer-crash');- const storage = new MemoryStorage({- localDataDirectory: tmpLocation,- persistStorage: false,- });+describe('MemoryStorageClient should not crash when saving a big buffer', () => {+ const storage = new MemoryStorageClient();let store: KeyValueStoreClient;@@ -20,10 +13,6 @@ describe('MemoryStorage should not crash when saving a big buffer', () => {store = await storage.createKeyValueStoreClient();});- afterAll(async () => {- await rm(tmpLocation, { force: true, recursive: true });- });-test('should not crash when saving a big buffer', async () => {let zip: Buffer;diff --git a/packages/memory-storage/test/no-writing-to-disk.test.ts b/packages/memory-storage/test/no-writing-to-disk.test.tsdeleted file mode 100644index 9006957a2..000000000--- a/packages/memory-storage/test/no-writing-to-disk.test.ts+++ /dev/null@@ -1,54 +0,0 @@-import { readdir, rm } from 'node:fs/promises';-import { resolve } from 'node:path';--import { MemoryStorage } from '@crawlee/memory-storage';--import { waitTillWrittenToDisk } from './__shared__.js';--describe('persistStorage option', () => {- const tmpLocation = resolve(import.meta.dirname, './tmp/no-writing-to-disk');-- afterAll(async () => {- await rm(tmpLocation, { force: true, recursive: true });- });-- describe('when false and writeMetadata is also false', () => {- const localDataDirectory = resolve(tmpLocation, './no-metadata');- const storage = new MemoryStorage({- localDataDirectory,- persistStorage: false,- });-- test('creating a key-value pair in a key-value store should not write data to the disk', async () => {- const keyValueStore = await storage.createKeyValueStoreClient();- await keyValueStore.setValue({ key: 'foo', value: 'test' });-- // We check that reading the directory for the store throws an error, which means it wasn't created on disk- await expect(async () => readdir(localDataDirectory)).rejects.toThrow();- });- });-- describe('when false and writeMetadata is true', () => {- const localDataDirectory = resolve(tmpLocation, './with-metadata');- const storage = new MemoryStorage({- localDataDirectory,- persistStorage: false,- writeMetadata: true,- });-- test('creating a key-value pair in a key-value store should not write data to the disk, but it should write the __metadata__ file', async () => {- const keyValueStore = await storage.createKeyValueStoreClient();- await keyValueStore.setValue({ key: 'foo', value: 'test' });-- const keyValueStoreInfo = await keyValueStore.getMetadata();- const storePath = resolve(storage.keyValueStoresDirectory, keyValueStoreInfo.id);-- await waitTillWrittenToDisk(storePath);-- const directoryFiles = await readdir(storePath);-- expect(directoryFiles).toHaveLength(1);- expect(directoryFiles).toEqual(['__metadata__.json']);- });- });-});diff --git a/packages/memory-storage/test/request-queue/forefront.test.ts b/packages/memory-storage/test/request-queue/forefront.test.tsindex 0553d87c4..bdb9b16d5 100644--- a/packages/memory-storage/test/request-queue/forefront.test.ts+++ b/packages/memory-storage/test/request-queue/forefront.test.ts@@ -2,7 +2,7 @@ import { rm } from 'node:fs/promises';import { resolve } from 'node:path';import { setTimeout as sleep } from 'node:timers/promises';-import { MemoryStorage } from '@crawlee/memory-storage';+import { MemoryStorageClient } from '@crawlee/memory-storage';import type { RequestQueueClient } from '@crawlee/types';/**@@ -21,9 +21,7 @@ async function fetchOrder(client: RequestQueueClient): Promise<string[]> {}describe('RequestQueue respects `forefront` when fetching requests', () => {- const storage = new MemoryStorage({- persistStorage: false,- });+ const storage = new MemoryStorageClient();let requestQueue: RequestQueueClient;@@ -147,122 +145,33 @@ describe('RequestQueue respects `forefront` when fetching requests', () => {});});-describe('RequestQueue locks fetched requests', () => {- const storage = new MemoryStorage({ persistStorage: false });+describe('RequestQueue holds fetched requests in progress', () => {+ const storage = new MemoryStorageClient();let requestQueue: RequestQueueClient;beforeEach(async () => {- requestQueue = await storage.createRequestQueueClient({ name: 'locking' });+ requestQueue = await storage.createRequestQueueClient({ name: 'in-progress' });});afterEach(async () => {await requestQueue.drop();});- test('a fetched request becomes available again after its lock expires', async () => {- vitest.useFakeTimers();-- try {- await requestQueue.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }]);-- const first = await requestQueue.fetchNextRequest();- expect(first!.uniqueKey).toBe('1');-- // While locked, the request is not handed out again.- expect(await requestQueue.fetchNextRequest()).toBeNull();-- // After the lock expires (default 3 minutes), the request is fetchable again — this is what- // prevents a crashed consumer from blocking its requests forever.- vitest.advanceTimersByTime(3 * 60 * 1000 + 1000);-- const retried = await requestQueue.fetchNextRequest();- expect(retried!.uniqueKey).toBe('1');- } finally {- vitest.useRealTimers();- }- });-});--describe('RequestQueue locking is visible across clients sharing on-disk storage', () => {- const tmpLocation = resolve(import.meta.dirname, './tmp/req-queue-cross-process');- // Two independent storage instances over the same directory emulate two separate processes.- const storageA = new MemoryStorage({ localDataDirectory: tmpLocation, persistStorage: true });- const storageB = new MemoryStorage({ localDataDirectory: tmpLocation, persistStorage: true });-- afterAll(async () => {- await rm(tmpLocation, { force: true, recursive: true });- });-- test('two clients on the same queue never fetch the same request', async () => {- const clientA = await storageA.createRequestQueueClient({ name: 'shared' });- await clientA.addBatchOfRequests([- { url: 'http://example.com/1', uniqueKey: '1' },- { url: 'http://example.com/2', uniqueKey: '2' },- ]);-- const clientB = await storageB.createRequestQueueClient({ name: 'shared' });-- const fromA = await clientA.fetchNextRequest();- const fromB = await clientB.fetchNextRequest();-- expect(fromA).not.toBeNull();- expect(fromB).not.toBeNull();- // The lock written by one client is observed by the other, so they get distinct requests.- expect(fromA!.uniqueKey).not.toBe(fromB!.uniqueKey);-- // Both requests are now locked, so neither client can fetch anything more.- expect(await clientA.fetchNextRequest()).toBeNull();- expect(await clientB.fetchNextRequest()).toBeNull();-- await clientA.drop();- });-- test('a client does not report the queue finished while another client holds the last request', async () => {- const clientA = await storageA.createRequestQueueClient({ name: 'shared-is-empty' });- await clientA.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }]);-- const clientB = await storageB.createRequestQueueClient({ name: 'shared-is-empty' });-- // Client A fetches (and thus locks) the only request.- const fromA = await clientA.fetchNextRequest();- expect(fromA).not.toBeNull();-- // Client B has nothing it can fetch right now, so from its point of view the queue is empty...- expect(await clientB.fetchNextRequest()).toBeNull();- expect(await clientB.isEmpty()).toBe(true);- // ...but the request still exists and is merely locked by A, so B must NOT consider the queue- // finished — otherwise the crawler driving B could shut down while A is still processing.- expect(await clientB.isFinished()).toBe(false);-- // Once A handles the request, it is gone for good and B sees a finished queue.- await clientA.markRequestAsHandled({ ...fromA!, id: fromA!.id! });- expect(await clientB.isEmpty()).toBe(true);- expect(await clientB.isFinished()).toBe(true);-- await clientA.drop();- });-- test('teardown releases this client locks so another client can fetch immediately', async () => {- const clientA = await storageA.createRequestQueueClient({ name: 'shared-teardown' });- await clientA.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }]);-- // Client A fetches (locks) the request, then the process tears down without handling it.- const fromA = await clientA.fetchNextRequest();- expect(fromA).not.toBeNull();+ test('a fetched request stays in progress and is only fetchable again once reclaimed', async () => {+ await requestQueue.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }]);- const clientB = await storageB.createRequestQueueClient({ name: 'shared-teardown' });- // While A holds the lock, B cannot fetch the request.- expect(await clientB.fetchNextRequest()).toBeNull();+ const first = await requestQueue.fetchNextRequest();+ expect(first!.uniqueKey).toBe('1');- // Tearing down A's storage releases its locks (instead of leaving them until the 3-minute- // expiry), so B can pick the request up right away.- await storageA.teardown();+ // While in progress, the request is not handed out again. The in-memory queue lives in a single+ // process, so there is no lock expiry — the request stays in progress until it is explicitly+ // reclaimed (or handled).+ expect(await requestQueue.fetchNextRequest()).toBeNull();- const fromB = await clientB.fetchNextRequest();- expect(fromB).not.toBeNull();- expect(fromB!.uniqueKey).toBe('1');+ await requestQueue.reclaimRequest({ ...first!, id: first!.id as string });- await clientB.drop();+ const retried = await requestQueue.fetchNextRequest();+ expect(retried!.uniqueKey).toBe('1');});});diff --git a/packages/memory-storage/test/request-queue/handledRequestCount-should-update.test.ts b/packages/memory-storage/test/request-queue/handledRequestCount-should-update.test.tsindex 07578ffce..2b8e56366 100644--- a/packages/memory-storage/test/request-queue/handledRequestCount-should-update.test.ts+++ b/packages/memory-storage/test/request-queue/handledRequestCount-should-update.test.ts@@ -1,10 +1,8 @@-import { MemoryStorage } from '@crawlee/memory-storage';+import { MemoryStorageClient } from '@crawlee/memory-storage';import type { RequestQueueClient } from '@crawlee/types';describe('RequestQueue handledRequestCount should update', () => {- const storage = new MemoryStorage({- persistStorage: false,- });+ const storage = new MemoryStorageClient();let requestQueue: RequestQueueClient;diff --git a/packages/memory-storage/test/request-queue/in-progress.test.ts b/packages/memory-storage/test/request-queue/in-progress.test.tsnew file mode 100644index 000000000..f9ce49c93--- /dev/null+++ b/packages/memory-storage/test/request-queue/in-progress.test.ts@@ -0,0 +1,99 @@+import { MemoryStorageClient } from '@crawlee/memory-storage';+import type { RequestQueueClient } from '@crawlee/types';++describe('RequestQueue in-progress requests', () => {+ test('a fetched request stays in progress until it is handled or reclaimed', async () => {+ const storage = new MemoryStorageClient();+ const queue: RequestQueueClient = await storage.createRequestQueueClient({ name: 'in-progress' });++ await queue.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }]);++ const fetched = await queue.fetchNextRequest();+ expect(fetched).not.toBeNull();++ // While in progress, the request must not be handed out again. There is nothing fetchable, so the+ // queue is empty — but the in-progress request means it is not yet finished. Unlike the previous+ // disk-backed implementation, there is no lock expiry: the request never becomes fetchable again+ // on its own.+ expect(await queue.fetchNextRequest()).toBeNull();+ expect(await queue.isEmpty()).toBe(true);+ expect(await queue.isFinished()).toBe(false);+ });++ test('a fetched request becomes fetchable again once reclaimed', async () => {+ const storage = new MemoryStorageClient();+ const queue: RequestQueueClient = await storage.createRequestQueueClient({ name: 'reclaim' });++ await queue.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }]);++ const fetched = await queue.fetchNextRequest();+ expect(fetched).not.toBeNull();++ const result = await queue.reclaimRequest({ ...fetched!, id: fetched!.id as string }, { forefront: true });+ expect(result).not.toBeNull();++ // The reclaimed request is no longer in progress, so it is pending and fetchable again.+ expect(await queue.isEmpty()).toBe(false);+ expect((await queue.fetchNextRequest())?.uniqueKey).toBe('1');+ });++ test('an in-progress request can be marked as handled', async () => {+ const storage = new MemoryStorageClient();+ const queue: RequestQueueClient = await storage.createRequestQueueClient({ name: 'handle' });++ await queue.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }]);++ const fetched = await queue.fetchNextRequest();+ expect(fetched).not.toBeNull();++ const result = await queue.markRequestAsHandled({ ...fetched!, id: fetched!.id as string });+ expect(result).not.toBeNull();++ // The request must be counted as handled and never handed out again.+ const metadata = await queue.getMetadata();+ expect(metadata.handledRequestCount).toBe(1);+ expect(metadata.pendingRequestCount).toBe(0);+ expect(await queue.isEmpty()).toBe(true);+ expect(await queue.isFinished()).toBe(true);+ expect(await queue.fetchNextRequest()).toBeNull();+ });++ test('multiple requests are each handed out only once while in progress', async () => {+ const storage = new MemoryStorageClient();+ const queue: RequestQueueClient = await storage.createRequestQueueClient({ name: 'multi' });++ await queue.addBatchOfRequests([+ { url: 'http://example.com/1', uniqueKey: '1' },+ { url: 'http://example.com/2', uniqueKey: '2' },+ ]);++ const first = await queue.fetchNextRequest();+ const second = await queue.fetchNextRequest();++ expect(first).not.toBeNull();+ expect(second).not.toBeNull();+ // The two fetches return distinct requests; neither is handed out twice.+ expect(first!.uniqueKey).not.toBe(second!.uniqueKey);++ // Both are now in progress, so nothing more is fetchable.+ expect(await queue.fetchNextRequest()).toBeNull();+ expect(await queue.isFinished()).toBe(false);+ });++ test('dropping a queue with a pending forefront request does not corrupt later head scans', async () => {+ const storage = new MemoryStorageClient();+ const queue: RequestQueueClient = await storage.createRequestQueueClient({ name: 'drop-forefront' });++ // A forefront request leaves an id in `forefrontRequestIds`. `drop` must clear that alongside the+ // `requests` map, otherwise a later head scan would resolve the dangling id to a missing request+ // and dereference `undefined`.+ await queue.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }], { forefront: true });++ await queue.drop();++ // A head scan on the dropped client must not throw and must report an empty, finished queue.+ await expect(queue.isEmpty()).resolves.toBe(true);+ await expect(queue.isFinished()).resolves.toBe(true);+ await expect(queue.fetchNextRequest()).resolves.toBeNull();+ });+});diff --git a/packages/memory-storage/test/request-queue/request-lock-secs.test.ts b/packages/memory-storage/test/request-queue/request-lock-secs.test.tsdeleted file mode 100644index 4b9d55ea4..000000000--- a/packages/memory-storage/test/request-queue/request-lock-secs.test.ts+++ /dev/null@@ -1,102 +0,0 @@-import { MemoryStorage } from '@crawlee/memory-storage';-import type { RequestQueueClient } from '@crawlee/types';--describe('RequestQueue request lock duration', () => {- beforeEach(() => {- vitest.useFakeTimers();- });-- afterEach(() => {- vitest.useRealTimers();- });-- test('a fetched request stays locked for the default 3 minutes, then becomes fetchable again', async () => {- const storage = new MemoryStorage({ persistStorage: false });- const queue: RequestQueueClient = await storage.createRequestQueueClient({ name: 'default-lock' });-- await queue.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }]);-- const fetched = await queue.fetchNextRequest();- expect(fetched).not.toBeNull();-- // While the lock is held, the request must not be handed out again. There is nothing fetchable,- // so the queue is empty — but the locked request means it is not yet finished.- expect(await queue.fetchNextRequest()).toBeNull();- expect(await queue.isEmpty()).toBe(true);- expect(await queue.isFinished()).toBe(false);-- // Just before the default lock expires, it is still locked.- vitest.advanceTimersByTime(3 * 60 * 1000 - 1000);- expect(await queue.fetchNextRequest()).toBeNull();-- // Once the lock expires, the same request is fetchable again.- vitest.advanceTimersByTime(2 * 1000);- const refetched = await queue.fetchNextRequest();- expect(refetched).not.toBeNull();- expect(refetched!.uniqueKey).toBe('1');- });-- test('setExpectedRequestProcessingTimeSecs changes how long a fetched request stays locked', async () => {- const storage = new MemoryStorage({ persistStorage: false });- const queue: RequestQueueClient = await storage.createRequestQueueClient({ name: 'set-lock-secs' });-- // Raise the lock well above the 3-minute default — expect requests to stay locked for 10 minutes.- queue.setExpectedRequestProcessingTimeSecs!(600);-- await queue.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }]);- expect(await queue.fetchNextRequest()).not.toBeNull();-- // After the default 3-minute lock would have expired, the request is still locked.- vitest.advanceTimersByTime(4 * 60 * 1000);- expect(await queue.fetchNextRequest()).toBeNull();-- // Only after the configured 10-minute lock expires is it fetchable again.- vitest.advanceTimersByTime(6 * 60 * 1000 + 1000);- expect((await queue.fetchNextRequest())?.uniqueKey).toBe('1');- });-- test('a request can still be marked as handled after its lock has expired', async () => {- const storage = new MemoryStorage({ persistStorage: false });- const queue: RequestQueueClient = await storage.createRequestQueueClient({ name: 'handle-after-expiry' });-- await queue.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }]);-- const fetched = await queue.fetchNextRequest();- expect(fetched).not.toBeNull();-- // The consumer takes longer than the lock duration to process the request.- vitest.advanceTimersByTime(3 * 60 * 1000 + 1000);-- // Marking it handled must still succeed — a slow consumer should not be penalized.- const result = await queue.markRequestAsHandled({ ...fetched!, id: fetched!.id as string });- expect(result).not.toBeNull();-- // The request must be counted as handled and never handed out again.- const metadata = await queue.getMetadata();- expect(metadata.handledRequestCount).toBe(1);- expect(metadata.pendingRequestCount).toBe(0);- expect(await queue.isEmpty()).toBe(true);- expect(await queue.isFinished()).toBe(true);- expect(await queue.fetchNextRequest()).toBeNull();- });-- test('a request can still be reclaimed after its lock has expired', async () => {- const storage = new MemoryStorage({ persistStorage: false });- const queue: RequestQueueClient = await storage.createRequestQueueClient({ name: 'reclaim-after-expiry' });-- await queue.addBatchOfRequests([{ url: 'http://example.com/1', uniqueKey: '1' }]);-- const fetched = await queue.fetchNextRequest();- expect(fetched).not.toBeNull();-- // The consumer fails after holding the request longer than the lock duration.- vitest.advanceTimersByTime(3 * 60 * 1000 + 1000);-- const result = await queue.reclaimRequest({ ...fetched!, id: fetched!.id as string }, { forefront: true });- expect(result).not.toBeNull();-- // The reclaimed request is still pending and fetchable again.- expect(await queue.isEmpty()).toBe(false);- expect((await queue.fetchNextRequest())?.uniqueKey).toBe('1');- });-});diff --git a/packages/memory-storage/test/reverse-datataset-list.test.ts b/packages/memory-storage/test/reverse-datataset-list.test.tsindex 28b0388c8..8f42cb6df 100644--- a/packages/memory-storage/test/reverse-datataset-list.test.ts+++ b/packages/memory-storage/test/reverse-datataset-list.test.ts@@ -1,24 +1,13 @@-import { rm } from 'node:fs/promises';-import { resolve } from 'node:path';--import { MemoryStorage } from '@crawlee/memory-storage';+import { MemoryStorageClient } from '@crawlee/memory-storage';import type { DatasetClient } from '@crawlee/types';const elements = Array.from({ length: 10 }, (_, i) => ({ number: i }));describe('Dataset#getData respects the desc option', () => {- const localDataDirectory = resolve(import.meta.dirname, './tmp/desc');- const storage = new MemoryStorage({- localDataDirectory,- persistStorage: false,- });+ const storage = new MemoryStorageClient();let dataset: DatasetClient;- afterAll(async () => {- await rm(localDataDirectory, { force: true, recursive: true });- });-beforeAll(async () => {dataset = await storage.createDatasetClient({ name: 'false' });diff --git a/packages/types/src/storages.ts b/packages/types/src/storages.tsindex 6c0da3cdf..1bda98559 100644--- a/packages/types/src/storages.ts+++ b/packages/types/src/storages.ts@@ -430,7 +430,7 @@ export interface StorageClient {** The key is used by `StorageInstanceManager` to partition the storage cache per-backend,* so that two storages with the same name but backed by different clients- * (e.g. a local `MemoryStorage` and a cloud `ApifyClient`) are cached as separate instances.+ * (e.g. a local `MemoryStorageClient` and a cloud `ApifyClient`) are cached as separate instances.** When not provided, the fallback uses the client's constructor name, so different* `StorageClient` implementations automatically get separate cache partitions.diff --git a/pnpm-lock.yaml b/pnpm-lock.yamlindex 2c9cf535b..d7dcaa39f 100644--- a/pnpm-lock.yaml+++ b/pnpm-lock.yaml@@ -427,6 +427,9 @@ importers:'@apify/utilities':specifier: ^2.15.5version: 2.27.0+ '@crawlee/fs-storage':+ specifier: workspace:*+ version: link:../fs-storage'@crawlee/memory-storage':specifier: workspace:*version: link:../memory-storage@@ -493,6 +496,9 @@ importers:'@crawlee/core':specifier: workspace:*version: link:../core+ '@crawlee/fs-storage':+ specifier: workspace:*+ version: link:../fs-storage'@crawlee/http':specifier: workspace:*version: link:../http-crawler@@ -502,6 +508,9 @@ importers:'@crawlee/linkedom':specifier: workspace:*version: link:../linkedom-crawler+ '@crawlee/memory-storage':+ specifier: workspace:*+ version: link:../memory-storage'@crawlee/playwright':specifier: workspace:*version: link:../playwright-crawler@@ -527,6 +536,36 @@ importers:specifier: ^2.8.1version: 2.8.1+ packages/fs-storage:+ dependencies:+ '@crawlee/types':+ specifier: workspace:*+ version: link:../types+ '@sapphire/async-queue':+ specifier: ^1.5.5+ version: 1.5.5+ '@sapphire/shapeshift':+ specifier: ^4.0.0+ version: 4.0.0+ content-type:+ specifier: ^1.0.5+ version: 1.0.5+ fs-extra:+ specifier: ^11.3.0+ version: 11.3.4+ json5:+ specifier: ^2.2.3+ version: 2.2.3+ mime-types:+ specifier: ^3.0.1+ version: 3.0.2+ proper-lockfile:+ specifier: ^4.1.2+ version: 4.1.2+ tslib:+ specifier: ^2.8.1+ version: 2.8.1+packages/got-scraping-client:dependencies:'@crawlee/http-client':@@ -688,21 +727,12 @@ importers:content-type:specifier: ^1.0.5version: 1.0.5- fs-extra:- specifier: ^11.3.0- version: 11.3.4json5:specifier: ^2.2.3version: 2.2.3mime-types:specifier: ^3.0.1version: 3.0.2- p-limit:- specifier: ^6.2.0- version: 6.2.0- proper-lockfile:- specifier: ^4.1.2- version: 4.1.2tslib:specifier: ^2.8.1version: 2.8.1diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yamlindex 9b4040753..fa4868bc9 100644--- a/pnpm-workspace.yaml+++ b/pnpm-workspace.yaml@@ -1,31 +1,31 @@packages:- - packages/*- - docs- - website+ - packages/*+ - docs+ - websiteminimumReleaseAge: 1440minimumReleaseAgeExclude:- - "@apify/*"- - "@crawlee/*"- - "apify-client"- - "apify"- - "crawlee"- - "got-scraping"+ - "@apify/*"+ - "@crawlee/*"+ - "apify-client"+ - "apify"+ - "crawlee"+ - "got-scraping"overrides:- playwright-core: 1.61.1- "@browserbasehq/stagehand": 3.0.7- # Dedup minimatch to v9 everywhere except inside lerna — lerna 9.x's bundled- # code (`__toESM(require('minimatch')).default(...)`) only works with v3,- # whose CJS export *is* the function. Pinning v9 there caused the publish- # step to silently report 0 changed packages.- minimatch: "^9.0.0"- tmp: "^0.2.7"- "@puppeteer/browsers": "^3.0.4"- form-data: "^4.0.6"- tar: "^7.5.16"- "lerna>js-yaml": "^4.2.0"- "lerna>minimatch": "^3.1.4"+ playwright-core: 1.61.1+ "@browserbasehq/stagehand": 3.0.7+ # Dedup minimatch to v9 everywhere except inside lerna — lerna 9.x's bundled+ # code (`__toESM(require('minimatch')).default(...)`) only works with v3,+ # whose CJS export *is* the function. Pinning v9 there caused the publish+ # step to silently report 0 changed packages.+ minimatch: "^9.0.0"+ tmp: "^0.2.7"+ "@puppeteer/browsers": "^3.0.4"+ form-data: "^4.0.6"+ tar: "^7.5.16"+ "lerna>js-yaml": "^4.2.0"+ "lerna>minimatch": "^3.1.4"# pnpm 11 replaces `onlyBuiltDependencies` with an explicit `allowBuilds` map.# Each entry must be true (build allowed) or false (build skipped) — pnpm 11@@ -33,19 +33,20 @@ overrides:# (combined with `strictDepBuilds: false` below so the install can still# proceed when new build-requesting deps appear without a manual entry).allowBuilds:- "@apify/ui-icons": true- "@playwright/browser-chromium": true- "@playwright/browser-firefox": true- "@playwright/browser-webkit": true- "@swc/core": true- better-sqlite3: true- bufferutil: true- core-js: true- esbuild: true- nx: true- protobufjs: true- puppeteer: true- unrs-resolver: true+ "@apify/ui-icons": true+ "@playwright/browser-chromium": true+ "@playwright/browser-firefox": true+ "@playwright/browser-webkit": true+ "@swc/core": true+ better-sqlite3: true+ bufferutil: true+ core-js: true+ core-js-pure: set this to true or false+ esbuild: true+ nx: true+ protobufjs: true+ puppeteer: true+ unrs-resolver: truestrictDepBuilds: false@@ -64,8 +65,8 @@ nodeLinker: hoistedlinkWorkspacePackages: truepreferWorkspacePackages: truepublicHoistPattern:- - "*"+ - "*"patchedDependencies:- "@docusaurus/[email protected]": patches/@[email protected]- "@signalwire/[email protected]": patches/@signalwire+docusaurus-plugin-llms-txt+1.2.2.patch+ "@docusaurus/[email protected]": patches/@[email protected]+ "@signalwire/[email protected]": patches/@signalwire+docusaurus-plugin-llms-txt+1.2.2.patchdiff --git a/test/core/crawlers/basic_crawler.test.ts b/test/core/crawlers/basic_crawler.test.tsindex 5854a3ad7..d7d3cb023 100644--- a/test/core/crawlers/basic_crawler.test.ts+++ b/test/core/crawlers/basic_crawler.test.ts@@ -23,7 +23,7 @@ import {SessionPool,} from '@crawlee/basic';import { RequestState, SessionPool, Statistics } from '@crawlee/core';-import { MemoryStorage } from '@crawlee/memory-storage';+import { MemoryStorageClient } from '@crawlee/memory-storage';import type { ISession, ProxyInfo } from '@crawlee/types';import type { Dictionary } from '@crawlee/utils';import { RobotsTxtFile, sleep } from '@crawlee/utils';@@ -2626,10 +2626,10 @@ describe('BasicCrawler', () => {});test("Crawlers with different storage clients don't share Datasets", async () => {- // Each crawler gets its own MemoryStorage with a different localDataDirectory,- // producing different clientCacheKeys and thus separate cache partitions.- const storageA = new MemoryStorage({ persistStorage: false, localDataDirectory: `${tmpDir}/storageA` });- const storageB = new MemoryStorage({ persistStorage: false, localDataDirectory: `${tmpDir}/storageB` });+ // Each crawler gets its own MemoryStorageClient instance; every instance has a unique+ // per-instance cache key, so they end up in separate cache partitions.+ const storageA = new MemoryStorageClient();+ const storageB = new MemoryStorageClient();const crawlerA = new BasicCrawler({ storageClient: storageA });const crawlerB = new BasicCrawler({ storageClient: storageB });@@ -2643,8 +2643,8 @@ describe('BasicCrawler', () => {});test('Crawlers with different storage clients run separately', async () => {- const storageA = new MemoryStorage({ persistStorage: false, localDataDirectory: `${tmpDir}/storageA` });- const storageB = new MemoryStorage({ persistStorage: false, localDataDirectory: `${tmpDir}/storageB` });+ const storageA = new MemoryStorageClient();+ const storageB = new MemoryStorageClient();const crawlerA = new BasicCrawler({requestHandler: () => {},diff --git a/test/core/storages/request_queue.test.ts b/test/core/storages/request_queue.test.tsindex 86b1a1bc0..f3f019164 100644--- a/test/core/storages/request_queue.test.ts+++ b/test/core/storages/request_queue.test.ts@@ -377,7 +377,10 @@ describe('RequestQueue remote', () => {describe('setExpectedRequestProcessingTimeSecs', () => {test('forwards the value to the client, but only ever raises it', async () => {const queue = await createRequestQueue();- const spy = vitest.spyOn(queue.client, 'setExpectedRequestProcessingTimeSecs');+ // The in-memory client does not implement this optional hint (it has no request locking to+ // tune), so attach a stub to verify the frontend's raise-only forwarding logic in isolation.+ const spy = vitest.fn();+ (queue.client as any).setExpectedRequestProcessingTimeSecs = spy;// First hint is forwarded.queue.setExpectedRequestProcessingTimeSecs(60);diff --git a/test/core/storages/storage_aliases.test.ts b/test/core/storages/storage_aliases.test.tsindex 919292f3a..907cb2bf0 100644--- a/test/core/storages/storage_aliases.test.ts+++ b/test/core/storages/storage_aliases.test.ts@@ -1,4 +1,11 @@-import { Dataset, KeyValueStore, RequestQueue } from '@crawlee/core';+import { resolve } from 'node:path';++import { FileSystemStorageClient } from '@crawlee/fs-storage';+import { Dataset, KeyValueStore, RequestQueue, serviceLocator } from '@crawlee/core';+import { ensureDir, rm } from 'fs-extra';++import { cryptoRandomObjectId } from '@apify/utilities';+import { MemoryStorageEmulator } from '../../shared/MemoryStorageEmulator.js';const localStorageEmulator = new MemoryStorageEmulator();@@ -142,14 +149,17 @@ describe('storage aliases', () => {});describe('string identifier vs alias conflict (persistent storage)', () => {- const persistentEmulator = new MemoryStorageEmulator();+ const localStorageDir = resolve(import.meta.dirname, '..', 'tmp', 'fs-aliases', cryptoRandomObjectId(10));beforeEach(async () => {- await persistentEmulator.init({ persistStorage: true });+ serviceLocator.reset();+ await ensureDir(localStorageDir);+ serviceLocator.setStorageClient(new FileSystemStorageClient({ localDataDirectory: localStorageDir }));});afterAll(async () => {- await persistentEmulator.destroy();+ await rm(localStorageDir, { force: true, recursive: true });+ serviceLocator.getStorageInstanceManager().clearCache();});test('should throw when opening a string identifier that matches an existing alias on disk', async () => {diff --git a/test/shared/MemoryStorageEmulator.ts b/test/shared/MemoryStorageEmulator.tsindex d244ec672..7660ba00d 100644--- a/test/shared/MemoryStorageEmulator.ts+++ b/test/shared/MemoryStorageEmulator.ts@@ -1,6 +1,6 @@import { resolve } from 'node:path';-import { MemoryStorage } from '@crawlee/memory-storage';+import { MemoryStorageClient } from '@crawlee/memory-storage';import { Configuration, serviceLocator } from 'crawlee';import { ensureDir } from 'fs-extra';@@ -12,7 +12,7 @@ import { StorageEmulator } from './StorageEmulator.js';const LOCAL_EMULATION_DIR = resolve(import.meta.dirname, '..', 'tmp', 'memory-emulation-dir');export class MemoryStorageEmulator extends StorageEmulator {- private storage!: MemoryStorage;+ private storage!: MemoryStorageClient;override async init({ dirName = cryptoRandomObjectId(10), persistStorage = false }: MemoryEmulatorOptions = {}) {await super.init();@@ -20,7 +20,11 @@ export class MemoryStorageEmulator extends StorageEmulator {this.localStorageDirectories.push(localStorageDir);await ensureDir(localStorageDir);- this.storage = new MemoryStorage({ localDataDirectory: localStorageDir, persistStorage, writeMetadata: false });+ // `MemoryStorageClient` is purely in-memory and ignores any disk-related options, so they are+ // not passed here. `persistStorage` is accepted for API compatibility with the emulator options+ // but has no effect on the in-memory storage.+ void persistStorage;+ this.storage = new MemoryStorageClient();serviceLocator.setStorageClient(this.storage);log.debug(`Initialized emulated memory storage in folder ${localStorageDir}`);diff --git a/website/docusaurus.config.js b/website/docusaurus.config.jsindex 9e41a6cd0..46b6e0192 100644--- a/website/docusaurus.config.js+++ b/website/docusaurus.config.js@@ -16,6 +16,7 @@ const packages = ['linkedom-crawler','stagehand-crawler','memory-storage',+ 'fs-storage','utils','types','impit-client',@@ -34,6 +35,7 @@ const packagesOrder = ['@crawlee/http-client','@crawlee/browser','@crawlee/memory-storage',+ '@crawlee/fs-storage','@crawlee/browser-pool','@crawlee/utils','@crawlee/types',