Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
feat!: Transactional storage access (#3953)
Jan Bucharbb6fc1d2c633
diff --git a/docs/guides/result_storage.mdx b/docs/guides/result_storage.mdxindex eb2b4ac50..b2354166b 100644--- a/docs/guides/result_storage.mdx+++ b/docs/guides/result_storage.mdx@@ -111,3 +111,83 @@ await purgeDefaultStorages();```Calling this function will clean up the default results storage directories except the `INPUT` key in default key-value store directory. This is a shortcut for running (optional) `purge` method on the <ApiLink to="core/interface/StorageBackend">`StorageBackend`</ApiLink> interface, in other words it will call the `purge` method of the underlying storage implementation we are currently using. In addition, this method will make sure the storage is purged only once for a given execution context, so it is safe to call it multiple times.++## Transactional storage++A request handler either finishes, or it doesn't. Without further care, a handler that pushes a few dataset items, updates a key-value record and *then* throws would leave those writes behind — and when the request is retried, they would happen again, duplicating your data.++Crawlee prevents that by wrapping every request in a **storage transaction**. Writes made through the storage classes (<ApiLink to="core/class/Dataset">`Dataset`</ApiLink>, <ApiLink to="core/class/KeyValueStore">`KeyValueStore`</ApiLink>, <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>) and the context helpers (`pushData`, `enqueueLinks`, ...) while a request is being handled are recorded rather than applied. When the request handler succeeds, they are replayed into real storage all together; when it throws, they are dropped. A retry therefore never double-writes, and a failed request leaves no partial results behind.++This is **on by default** for every crawler and covers the whole request lifecycle — `preNavigationHooks`, `postNavigationHooks`, `extendContext` and the request handler alike.++### What you can rely on++- **Atomicity per request.** A request's storage writes become visible all together on success, or not at all on failure.+- **Read-your-own-writes.** A read that follows a write in the same handler sees the written value — `getData()`, `getValue()`, key listings, dataset iteration and `getInfo()` all merge the handler's buffered writes with the real storage contents.+- **Isolation of uncommitted writes.** One handler's uncommitted writes are invisible to concurrently running handlers — to pass data between handlers, use `useState()` instead of the key-value store.+- **Fidelity.** Values are captured at write time (via `structuredClone`), so mutating an object after passing it to `pushData()` or `setValue()` affects neither what the handler reads back nor what gets committed.++### What the transaction deliberately does *not* cover++- **Request queue additions (by default).** Adding requests is idempotent (the queue deduplicates by `uniqueKey`), and buffering them would starve the crawl of new work until the handler finishes. New requests are therefore applied immediately (the `writeThrough` policy) and are *not* rolled back when the handler fails. If you need strict all-or-nothing enqueues, opt into buffering:++ ```typescript+ const crawler = new CheerioCrawler({+ transactionalStorage: { requestQueue: 'deferred' },+ // ...+ });+ ```++ Under the `deferred` policy, the `requestId` returned by an `addRequests()` call inside a handler is provisional — the real id is assigned by the storage backend when the transaction commits.++- **Shared state.** The object returned by `useState()` / <ApiLink to="core/class/KeyValueStore#getAutoSavedValue">`KeyValueStore.getAutoSavedValue()`</ApiLink> is the sanctioned live channel shared by all handlers, and it stays live: mutations of it are **not** rolled back when a handler fails.++- **Cross-storage atomicity.** The commit spans multiple storages and multiple calls; delivery is *at-least-once*. If a commit fails partway, the request fails and is retried, and the retry may re-apply what already landed.++- **Error handlers.** `errorHandler` and `failedRequestHandler` run after the failed request's transaction has been rolled back, so their writes go straight to real storage — that is what error handlers are for.++- **Deferred cleanups.** Callbacks registered with `registerDeferredCleanup()` run after the transaction is closed, so their writes land immediately and are not rolled back. In <ApiLink to="playwright-crawler/class/AdaptivePlaywrightCrawler">`AdaptivePlaywrightCrawler`</ApiLink> they run once per request handler attempt, so a write there can land twice for one request — push your results from the request handler instead.++### Escape hatches++The feature is escapable at three granularities:++- **Whole feature:** `transactionalStorage: false` on the crawler options disables the mechanism entirely; every storage call behaves exactly as if no transactions existed. (Not supported by <ApiLink to="playwright-crawler/class/AdaptivePlaywrightCrawler">`AdaptivePlaywrightCrawler`</ApiLink>, which needs per-attempt buffering to work at all.)+- **Per storage type:** the `transactionalStorage: { requestQueue: 'writeThrough' | 'deferred' }` policy object, described above.+- **Per call site:** <ApiLink to="core/function/withDirectStorageAccess">`withDirectStorageAccess()`</ApiLink> runs a callback outside the transaction, so its writes land immediately and are never rolled back. Use it for progress files, streaming output, and anything that genuinely must not wait for the handler to succeed:++ ```typescript+ import { withDirectStorageAccess } from 'crawlee';++ async function requestHandler({ request }) {+ await withDirectStorageAccess(async () => {+ const store = await KeyValueStore.open();+ await store.setValue(`progress-${request.id}`, { startedAt: new Date() });+ });+ // ... the rest of the handler is transactional as usual+ }+ ```++### Operations that throw inside a transaction++A few operations cannot be buffered, and silently letting them through would produce storage states that no rollback can undo. They throw inside a transaction, with an error pointing at `withDirectStorageAccess()`:++- `Dataset.drop()`, `KeyValueStore.drop()`, `RequestQueue.drop()` and `RequestQueue.purge()`,+- the request queue processing internals (`fetchNextRequest()`, `markRequestAsHandled()`, `reclaimRequest()`),+- `KeyValueStore.setValue()` with a **stream** value — a stream can only be consumed once, so it cannot serve both a read within the handler and the commit replay. Write streams under `withDirectStorageAccess()`.++### Programmatic use++Outside a crawler, storage is non-transactional unless you ask for it explicitly:++```typescript+import { withStorageTransaction, Dataset } from 'crawlee';++await withStorageTransaction(async () => {+ const dataset = await Dataset.open();+ await dataset.pushData({ some: 'data' }); // buffered+ // committed when the callback returns, rolled back if it throws+});+```++For full control (e.g. deciding whether to commit only after inspecting the results), use <ApiLink to="core/function/createStorageTransaction">`createStorageTransaction()`</ApiLink> and drive `run()` / `commit()` / `rollback()` / `dispose()` yourself. This is exactly what `AdaptivePlaywrightCrawler` does with its per-attempt transactions.diff --git a/docs/public-api/crawlee-basic.api.md b/docs/public-api/crawlee-basic.api.mdindex 7e5ba3119..c83e150c1 100644--- a/docs/public-api/crawlee-basic.api.md+++ b/docs/public-api/crawlee-basic.api.md@@ -50,6 +50,7 @@ import type { SkippedRequestCallback } from '@crawlee/core';import type { StatisticState } from '@crawlee/core';import type { StorageBackend } from '@crawlee/types';import type { StorageIdentifier } from '@crawlee/core';+import { StorageWritePolicy } from '@crawlee/core';import { StringPredicate } from 'ow';import type { TaskLoopPredicates } from '@crawlee/core';import { TimeoutError } from '@apify/timeout';@@ -141,6 +142,7 @@ export class BasicCrawler<Context extends CrawlingContext = CrawlingContext, ConblockedStatusCodes: ArrayPredicate<number>;retryOnBlocked: BooleanPredicate & BasePredicate<boolean | undefined>;respectRobotsTxtFile: AnyPredicate<boolean | object>;+ transactionalStorage: BasePredicate<boolean | Partial<StorageWritePolicy> | undefined>;onSkippedRequest: Predicate<Function> & BasePredicate<Function | undefined>;httpClient: ObjectPredicate<object> & BasePredicate<object | undefined>;configuration: ObjectPredicate<object> & BasePredicate<object | undefined>;@@ -221,6 +223,7 @@ export interface BasicCrawlerOptions<Context extends CrawlingContext = CrawlingCstatusMessageLoggingInterval?: number;storageBackend?: StorageBackend;taskLoopOptions?: TaskLoopPredicates;+ transactionalStorage?: boolean | Partial<StorageWritePolicy>;}// @public (undocumented)diff --git a/docs/public-api/crawlee-browser.api.md b/docs/public-api/crawlee-browser.api.mdindex aa7e4ea22..5fd79f7f0 100644--- a/docs/public-api/crawlee-browser.api.md+++ b/docs/public-api/crawlee-browser.api.md@@ -35,6 +35,7 @@ import { Predicate } from 'ow';import { Request as Request_2 } from '@crawlee/basic';import type { RequestHandler } from '@crawlee/basic';import type { RouterHandler } from '@crawlee/basic';+import { StorageWritePolicy } from '@crawlee/basic';import { StringPredicate } from 'ow';// Not exported by the entry point; reachable only as a referenced type.@@ -96,6 +97,7 @@ export abstract class BrowserCrawler<Page extends CommonPage = CommonPage, RespoblockedStatusCodes: ArrayPredicate<number>;retryOnBlocked: BooleanPredicate & BasePredicate<boolean | undefined>;respectRobotsTxtFile: AnyPredicate<boolean | object>;+ transactionalStorage: BasePredicate<boolean | Partial<StorageWritePolicy> | undefined>;onSkippedRequest: Predicate<Function> & BasePredicate<Function | undefined>;httpClient: ObjectPredicate<object> & BasePredicate<object | undefined>;configuration: ObjectPredicate<object> & BasePredicate<object | undefined>;diff --git a/docs/public-api/crawlee-core.api.md b/docs/public-api/crawlee-core.api.mdindex 432de615f..35768796e 100644--- a/docs/public-api/crawlee-core.api.md+++ b/docs/public-api/crawlee-core.api.md@@ -142,9 +142,6 @@ export interface CalculatedStatistics {requestTotalDurationMillis: number;}-// @public-export const checkStorageAccess: () => void | undefined;-// @public (undocumented)export interface ClientInfo {// (undocumented)@@ -344,6 +341,9 @@ export interface CreateSession {}): Session | Promise<Session>;}+// @public+export function createStorageTransaction(options?: StorageTransactionOptions): StorageTransaction;+// @publicexport class CriticalError extends NonRetryableError {}@@ -469,6 +469,17 @@ export interface DatasetExportToOptions extends DatasetExportOptions {export interface DatasetIteratorOptions extends Omit<DatasetDataOptions, 'offset' | 'limit' | 'clean' | 'skipHidden' | 'skipEmpty'> {}+// @public+export interface DatasetJournalEntry {+ items: Dictionary[];+ // (undocumented)+ recordedAt: Date;+ // (undocumented)+ storageId: string;+ // (undocumented)+ type: 'dataset';+}+// @publicexport interface DatasetMapper<Data, R> {(item: Data, index: number): Awaitable<R>;@@ -853,6 +864,20 @@ export interface IStorage {name?: string;}+// @public+export interface JournaledRequest {+ // (undocumented)+ label?: string;+ snapshot?: Dictionary;+ // (undocumented)+ uniqueKey: string;+ // (undocumented)+ url: string;+}++// @public (undocumented)+export type JournalEntry = DatasetJournalEntry | KeyValueStoreJournalEntry | RequestQueueJournalEntry;+// @publicexport interface KeyConsumer {// (undocumented)@@ -941,6 +966,19 @@ export interface KeyValueStoreIteratorOptions {prefix?: string;}+// @public+export interface KeyValueStoreJournalEntry {+ // (undocumented)+ key: string;+ // (undocumented)+ options?: RecordOptions;+ // (undocumented)+ storageId: string;+ // (undocumented)+ type: 'keyValueStore';+ value: unknown;+}+// @public (undocumented)export interface KeyValueStoreOptions {// (undocumented)@@ -1248,39 +1286,6 @@ export class RequestHandlerError extends Error {constructor(error: unknown, options?: ErrorOptions);}-// @public-export class RequestHandlerResult {- constructor(configuration: Configuration, crawleeStateKey: string);- // (undocumented)- addRequests: RestrictedCrawlingContext['addRequests'];- get calls(): ReadonlyDeep<{- pushData: Parameters<RestrictedCrawlingContext['pushData']>[];- addRequests: Parameters<RestrictedCrawlingContext['addRequests']>[];- }>;- get datasetItems(): ReadonlyDeep<{- item: Dictionary;- datasetIdentifier?: string | StorageIdentifier;- }[]>;- get enqueuedUrlLists(): ReadonlyDeep<{- listUrl: string;- label?: string;- }[]>;- get enqueuedUrls(): ReadonlyDeep<{- url: string;- label?: string;- }[]>;- // (undocumented)- getKeyValueStore: RestrictedCrawlingContext['getKeyValueStore'];- get keyValueStoreChanges(): ReadonlyDeep<Record<string, Record<string, {- changedValue: unknown;- options?: RecordOptions;- }>>>;- // (undocumented)- pushData: RestrictedCrawlingContext['pushData'];- // (undocumented)- useState: RestrictedCrawlingContext['useState'];-}-// @publicexport class RequestList implements IRequestLoader {// (undocumented)@@ -1465,6 +1470,17 @@ interface RequestQueueBackendOptions {storageBackend: MemoryStorageBackend;}+// @public+export interface RequestQueueJournalEntry {+ // (undocumented)+ forefront: boolean;+ // (undocumented)+ requests: JournaledRequest[];+ // (undocumented)+ type: 'requestQueue';+ writeThrough: boolean;+}+// @public (undocumented)export interface RequestQueueOperationInfo extends QueueOperationInfo {// (undocumented)@@ -2007,6 +2023,69 @@ export class StorageStatsTracker<T extends Record<keyof T, number>> {get current(): T;}+// @public+export class StorageTransaction implements StorageTransactionView {+ commit(): Promise<void>;+ // (undocumented)+ get datasetItems(): {+ item: Dictionary;+ datasetId: string;+ }[];+ dispose(): void;+ // (undocumented)+ get enqueuedUrls(): {+ url: string;+ label?: string;+ }[];+ get isActive(): boolean;+ readonly journal: JournalEntry[];+ // (undocumented)+ get keyValueStoreChanges(): Record<string, Record<string, {+ changedValue: unknown;+ options?: RecordOptions;+ }>>;+ readonly policy: StorageWritePolicy;+ rollback(): void;+ run<T>(callback: () => Awaitable<T>): Promise<T>;+ // (undocumented)+ get state(): StorageTransactionState;+}++// @public (undocumented)+export interface StorageTransactionOptions {+ commitTimeoutMillis?: number;+ policy?: Partial<StorageWritePolicy>;+}++// @public (undocumented)+export type StorageTransactionState = 'open' | 'committing' | 'committed' | 'failed' | 'rolledBack';++// @public+export interface StorageTransactionView {+ readonly datasetItems: {+ item: Dictionary;+ datasetId: string;+ }[];+ readonly enqueuedUrls: {+ url: string;+ label?: string;+ }[];+ readonly keyValueStoreChanges: Record<string, Record<string, {+ changedValue: unknown;+ options?: RecordOptions;+ }>>;+ // (undocumented)+ readonly state: StorageTransactionState;+}++// @public+export type StorageWriteMode = 'deferred' | 'writeThrough';++// @public+export interface StorageWritePolicy {+ requestQueue: StorageWriteMode;+}+// @publicexport interface SystemInfo {// (undocumented)@@ -2084,13 +2163,16 @@ export interface UseStateOptions {}// @public-export const withCheckedStorageAccess: <T>(checkFunction: () => void, callback: () => Awaitable<T>) => Promise<T>;+export function withDirectStorageAccess<T>(callback: () => Awaitable<T>): Promise<T>;// @public (undocumented)export type WithRequired<T, K extends keyof T> = T & {[P in K]-?: T[P];};+// @public+export function withStorageTransaction<T>(callback: (transaction: StorageTransaction) => Awaitable<T>, options?: StorageTransactionOptions): Promise<T>;+// (No @packageDocumentation comment for this package)```diff --git a/docs/public-api/crawlee-http.api.md b/docs/public-api/crawlee-http.api.mdindex f08ecdc5d..358b8fde0 100644--- a/docs/public-api/crawlee-http.api.md+++ b/docs/public-api/crawlee-http.api.md@@ -34,6 +34,7 @@ import { RouterHandler } from '@crawlee/basic';import { RouterRoutes } from '@crawlee/basic';import type { RouteSchemas } from '@crawlee/basic';import type { RoutesFromSchemas } from '@crawlee/basic';+import { StorageWritePolicy } from '@crawlee/basic';import { StringPredicate } from 'ow';import { Transform } from 'node:stream';@@ -146,6 +147,7 @@ export class HttpCrawler<Context extends InternalHttpCrawlingContext<any, any> =blockedStatusCodes: ArrayPredicate<number>;retryOnBlocked: BooleanPredicate & BasePredicate<boolean | undefined>;respectRobotsTxtFile: AnyPredicate<boolean | object>;+ transactionalStorage: BasePredicate<boolean | Partial<StorageWritePolicy> | undefined>;onSkippedRequest: Predicate<Function> & BasePredicate<Function | undefined>;httpClient: ObjectPredicate<object> & BasePredicate<object | undefined>;configuration: ObjectPredicate<object> & BasePredicate<object | undefined>;diff --git a/docs/public-api/crawlee-jsdom.api.md b/docs/public-api/crawlee-jsdom.api.mdindex 741b4fc56..7a8c41f03 100644--- a/docs/public-api/crawlee-jsdom.api.md+++ b/docs/public-api/crawlee-jsdom.api.md@@ -29,6 +29,7 @@ import type { RouterHandler } from '@crawlee/http';import type { RouterRoutes } from '@crawlee/http';import type { RouteSchemas } from '@crawlee/http';import type { RoutesFromSchemas } from '@crawlee/http';+import { StorageWritePolicy } from '@crawlee/http';import { StringPredicate } from 'ow';import { VirtualConsole } from 'jsdom';@@ -90,6 +91,7 @@ export class JSDOMCrawler<ContextExtension = Dictionary<never>, ExtendedContextblockedStatusCodes: ArrayPredicate<number>;retryOnBlocked: BooleanPredicate & BasePredicate<boolean | undefined>;respectRobotsTxtFile: AnyPredicate<boolean | object>;+ transactionalStorage: BasePredicate<boolean | Partial<StorageWritePolicy> | undefined>;onSkippedRequest: Predicate<Function> & BasePredicate<Function | undefined>;httpClient: ObjectPredicate<object> & BasePredicate<object | undefined>;configuration: ObjectPredicate<object> & BasePredicate<object | undefined>;diff --git a/docs/public-api/crawlee-playwright.api.md b/docs/public-api/crawlee-playwright.api.mdindex 9f8601c9f..f267e0481 100644--- a/docs/public-api/crawlee-playwright.api.md+++ b/docs/public-api/crawlee-playwright.api.md@@ -45,7 +45,6 @@ import type { RecoverableStatePersistenceOptions } from '@crawlee/core';import type { Request as Request_2 } from '@crawlee/core';import { Request as Request_3 } from '@crawlee/browser';import type { RequestHandler } from '@crawlee/browser';-import { RequestHandlerResult } from '@crawlee/core';import type { RequestTransform } from '@crawlee/browser';import type { Response as Response_2 } from 'playwright';import type { RouterHandler } from '@crawlee/browser';@@ -57,6 +56,8 @@ import type { SkippedRequestCallback } from '@crawlee/browser';import { Statistics } from '@crawlee/core';import type { StatisticsOptions } from '@crawlee/core';import type { StatisticState } from '@crawlee/core';+import type { StorageTransactionView } from '@crawlee/core';+import { StorageWritePolicy } from '@crawlee/browser';import { StringPredicate } from 'ow';import type { UrlPatternInput } from '@crawlee/browser';@@ -88,7 +89,6 @@ export class AdaptivePlaywrightCrawler<ContextExtension = Dictionary<never>, Extreadonly waitForSelector: AdaptivePlaywrightCrawlerContext["waitForSelector"];readonly parseWithCheerio: AdaptivePlaywrightCrawlerContext["parseWithCheerio"];}>;- protected getPendingRequestCountApproximation(): Promise<number>;// (undocumented)protected _init(): Promise<void>;// (undocumented)@@ -117,11 +117,10 @@ export interface AdaptivePlaywrightCrawlerContext<UserData extends Dictionary =export interface AdaptivePlaywrightCrawlerOptions<ContextExtension = Dictionary<never>, ExtendedContext extends AdaptivePlaywrightCrawlerContext = AdaptivePlaywrightCrawlerContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest_2<AdaptivePlaywrightCrawlerContext['request']>>> extends Omit<BasicCrawlerOptions<AdaptivePlaywrightCrawlerContext, ContextExtension, ExtendedContext, Routes>, 'preNavigationHooks' | 'postNavigationHooks'> {postNavigationHooks?: AdaptivePostNavigationHook<ContextExtension>[];preNavigationHooks?: AdaptiveHook<ContextExtension>[];- preventDirectStorageAccess?: boolean;renderingTypeDetectionRatio?: number;renderingTypePredictor?: IRenderingTypePredictor;- resultChecker?: (result: RequestHandlerResult) => boolean;- resultComparator?: (resultA: RequestHandlerResult, resultB: RequestHandlerResult) => boolean | 'equal' | 'different' | 'inconclusive';+ resultChecker?: (result: StorageTransactionView) => boolean;+ resultComparator?: (resultA: StorageTransactionView, resultB: StorageTransactionView) => boolean | 'equal' | 'different' | 'inconclusive';shouldPropagateError?: (error: Error, context: PlaywrightCrawlingContext) => Awaitable<boolean>;}@@ -230,7 +229,7 @@ interface EnqueueLinksByClickingElementsOptions {}// @public-export function fullResultComparator(resultA: RequestHandlerResult, resultB: RequestHandlerResult): boolean;+export function fullResultComparator(resultA: StorageTransactionView, resultB: StorageTransactionView): boolean;// @publicfunction gotoExtended(page: Page, request: Request_3, gotoOptions?: PlaywrightDirectNavigationOptions): Promise<Response_2 | null>;@@ -380,6 +379,7 @@ export class PlaywrightCrawler<ContextExtension = Dictionary<never>, ExtendedConblockedStatusCodes: ArrayPredicate<number>;retryOnBlocked: BooleanPredicate & BasePredicate<boolean | undefined>;respectRobotsTxtFile: AnyPredicate<boolean | object>;+ transactionalStorage: BasePredicate<boolean | Partial<StorageWritePolicy> | undefined>;onSkippedRequest: Predicate<Function> & BasePredicate<Function | undefined>;httpClient: ObjectPredicate<object> & BasePredicate<object | undefined>;configuration: ObjectPredicate<object> & BasePredicate<object | undefined>;diff --git a/docs/public-api/crawlee-puppeteer.api.md b/docs/public-api/crawlee-puppeteer.api.mdindex f26626465..526617c50 100644--- a/docs/public-api/crawlee-puppeteer.api.md+++ b/docs/public-api/crawlee-puppeteer.api.md@@ -40,6 +40,7 @@ import type { RouterRoutes } from '@crawlee/browser';import type { RouteSchemas } from '@crawlee/browser';import type { RoutesFromSchemas } from '@crawlee/browser';import type { SkippedRequestCallback } from '@crawlee/browser';+import { StorageWritePolicy } from '@crawlee/browser';import { StringPredicate } from 'ow';import type { Target } from 'puppeteer';import type { UrlPatternInput } from '@crawlee/browser';@@ -232,6 +233,7 @@ export class PuppeteerCrawler<ContextExtension = Dictionary<never>, ExtendedContblockedStatusCodes: ArrayPredicate<number>;retryOnBlocked: BooleanPredicate & BasePredicate<boolean | undefined>;respectRobotsTxtFile: AnyPredicate<boolean | object>;+ transactionalStorage: BasePredicate<boolean | Partial<StorageWritePolicy> | undefined>;onSkippedRequest: Predicate<Function> & BasePredicate<Function | undefined>;httpClient: ObjectPredicate<object> & BasePredicate<object | undefined>;configuration: ObjectPredicate<object> & BasePredicate<object | undefined>;diff --git a/docs/public-api/crawlee-stagehand.api.md b/docs/public-api/crawlee-stagehand.api.mdindex 75d59b0a8..887d34d18 100644--- a/docs/public-api/crawlee-stagehand.api.md+++ b/docs/public-api/crawlee-stagehand.api.md@@ -46,6 +46,7 @@ import type { RouterRoutes } from '@crawlee/browser';import type { RouteSchemas } from '@crawlee/browser';import type { RoutesFromSchemas } from '@crawlee/browser';import { Stagehand } from '@browserbasehq/stagehand';+import { StorageWritePolicy } from '@crawlee/browser';import type { StreamingAgentInstance } from '@browserbasehq/stagehand';import { StringPredicate } from 'ow';import type { z } from 'zod';@@ -120,6 +121,7 @@ export class StagehandCrawler<ContextExtension = Dictionary<never>, ExtendedContblockedStatusCodes: ArrayPredicate<number>;retryOnBlocked: BooleanPredicate & BasePredicate<boolean | undefined>;respectRobotsTxtFile: AnyPredicate<boolean | object>;+ transactionalStorage: BasePredicate<boolean | Partial<StorageWritePolicy> | undefined>;onSkippedRequest: Predicate<Function> & BasePredicate<Function | undefined>;httpClient: ObjectPredicate<object> & BasePredicate<object | undefined>;configuration: ObjectPredicate<object> & BasePredicate<object | undefined>;diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.mdindex dc7905254..98bf8d962 100644--- a/docs/upgrading/upgrading_v4.md+++ b/docs/upgrading/upgrading_v4.md@@ -23,6 +23,7 @@ This page summarizes the breaking changes in Crawlee v4. There are many, so the- **The session is the rotation unit.** A session carries its proxy, cookies and error score, and is rotated as a whole when blocked — replacing [proxy tiers](#tieredproxyurls-is-removed-from-proxyconfiguration) and [session rotation counters](#maxsessionrotations-and-requestsessionrotationcount-are-removed).- **Crawlers stop stepping on each other.** Multiple crawlers in one process [no longer share the default request queue](#multiple-crawler-instances-use-separate-default-request-queues), and repeated `run()` calls purge the queue instead of dropping and recreating it.- **Cookies behave.** `sendRequest` finally [respects your `Cookie` header](#cookie-handling-in-httpcrawler-and-sendrequest), and browser cookies set inside the handler are [persisted to the session](#browser-cookies-are-also-persisted-after-requesthandler).+- **No half-written results.** Storage writes in a request handler are [transactional](#storage-writes-in-request-handlers-are-transactional) — a handler that throws leaves nothing behind, and its retry does not duplicate data.- **Simpler storage backend contract.** A custom storage backend is now [4 classes instead of 7](#storagebackend-interface-simplified).## Rename cheat sheet@@ -543,6 +544,34 @@ const dataset = await Dataset.open();The same change applies to `CrawlingContext.getKeyValueStore()` and `CrawlingContext.pushData()` — both now accept `string | StorageIdentifier` for identifying the target storage.+### Storage writes in request handlers are transactional++Every crawler now wraps each request in a **storage transaction** (see the [Transactional storage](../guides/result-storage#transactional-storage) section of the Result Storage guide): storage writes made anywhere in the request lifecycle — hooks, `extendContext` and the request handler alike — are recorded and only applied to real storage when the request handler succeeds. A handler that throws leaves no partial writes behind, and a retry does not duplicate data.++The observable behavior of a *successful* handler is unchanged (reads within a handler see its own writes), but several things differ on the failure path and around handler boundaries:++- **Uncommitted writes are invisible to other handlers.** Using the key-value store as a live channel between concurrently running handlers no longer works — one handler's `setValue()` only becomes visible to others once its request succeeds. Use `useState()` for cross-handler communication.+- **`useState()` / `getAutoSavedValue()` are *not* transactional.** The shared state object stays live; mutations of it are not rolled back when a handler fails.+- **Request queue additions are applied immediately by default** (the `writeThrough` policy) and are not rolled back — deduplication by `uniqueKey` keeps retries idempotent. Pass `transactionalStorage: { requestQueue: 'deferred' }` for strict all-or-nothing enqueues.+- **Commit is at-least-once.** It spans multiple storages, so a commit that fails partway fails the request; the retry may re-apply writes that already landed.+- **`KeyValueStore.setValue()` with a stream value throws inside a request handler.** A stream can only be consumed once, so it cannot be buffered. Wrap the call in `withDirectStorageAccess()` to write it immediately:++ ```typescript+ import { withDirectStorageAccess } from 'crawlee';++ await withDirectStorageAccess(async () => keyValueStore.setValue('video', stream, { contentType: 'video/mp4' }));+ ```++- **`drop()`, `purge()` and the request queue processing internals throw inside a request handler**, since no rollback could undo them. `withDirectStorageAccess()` is the escape hatch there, too.+- **Key listing order changes inside a handler**: `keys()`, `values()` and `entries()` emit the handler's own (buffered) keys first, then the rest.++The mechanism can be disabled entirely with `transactionalStorage: false` on any crawler except `AdaptivePlaywrightCrawler` (which needs it to discard the writes of its losing request handler attempts).++#### Removed symbols and options++- `checkStorageAccess` and `withCheckedStorageAccess` are superseded by the transaction mechanism; the per-call-site helper is now `withDirectStorageAccess()`.+- The experimental `AdaptivePlaywrightCrawler` no longer needs its bespoke write-buffering machinery: the `preventDirectStorageAccess` option is gone (direct storage calls are now captured by the per-attempt transaction instead of throwing), and `RequestHandlerResult` is replaced by the read-only `StorageTransactionView`, which the `resultChecker` / `resultComparator` callbacks (and `fullResultComparator`) now receive. The view keeps the familiar accessors (`datasetItems`, `enqueuedUrls`, `keyValueStoreChanges`), so most callbacks only need a type change. The `calls` and `enqueuedUrlLists` accessors are gone — `requestsFromUrl` sources are now expanded when added, so the fetched URLs appear in `enqueuedUrls` (and are what `fullResultComparator` compares).+### `KeyValueStore.getPublicUrl` is now asyncThe `KeyValueStore.getPublicUrl` method is now asynchronous and reads the public URL directly from the storage backend.@@ -1696,6 +1725,7 @@ The full list of removed exports and members, for ctrl-F purposes. Where a repla- `HttpResponse`, `HttpResponseWithoutBody`, `StreamingHttpResponse`, `ResponseTypes`, `BaseHttpResponseData`, `SimpleHeaders`, `processHttpRequestOptions`, and `GotScrapingHttpClient` (from `@crawlee/core`) — the HTTP client surface moved to `@crawlee/http-client` / `@crawlee/got-scraping-client` (see [HTTP client packages and `BaseHttpClient` reshaped](#http-client-packages-and-basehttpclient-reshaped))- `StreamHandlerContext` and `FileDownloadOptions` types (from `@crawlee/http`) — see [`FileDownload` now extends `BasicCrawler`](#filedownload-now-extends-basiccrawler-and-no-longer-takes-filedownloadoptions)- `PlainResponse` type (from `@crawlee/http`) — it wrapped the `got-scraping` response and is gone along with the rest of the old HTTP response surface (see [`CrawlingContext.response` is now of type `Response`](#crawlingcontextresponse-is-now-of-type-response))+- `checkStorageAccess`, `withCheckedStorageAccess` and the `RequestHandlerResult` type — superseded by the storage transaction mechanism; use `withDirectStorageAccess()` and `StorageTransactionView` (see [Storage writes in request handlers are transactional](#storage-writes-in-request-handlers-are-transactional))#### The protected `BasicCrawler.crawlingContexts` map is removeddiff --git a/packages/basic-crawler/src/internals/basic-crawler.ts b/packages/basic-crawler/src/internals/basic-crawler.tsindex 81e17c4f0..266769b17 100644--- a/packages/basic-crawler/src/internals/basic-crawler.ts+++ b/packages/basic-crawler/src/internals/basic-crawler.ts@@ -29,6 +29,7 @@ import type {Source,StatisticState,StorageIdentifier,+ StorageWritePolicy,TaskLoopPredicates,TypedRequestsLike,} from '@crawlee/core';@@ -41,7 +42,9 @@ import {ContextPipelineCleanupError,ContextPipelineInitializationError,ContextPipelineInterruptedError,+ createStorageTransaction,CriticalError,+ currentStorageTransaction,Dataset,enqueueLinks,EnqueueStrategy,@@ -70,6 +73,7 @@ import {Statistics,validateUserData,validators,+ withDirectStorageAccess,} from '@crawlee/core';import { FetchHttpClient } from '@crawlee/http-client';import type {@@ -85,7 +89,7 @@ import type {} from '@crawlee/types';import { isAsyncIterable, isIterable, RobotsTxtFile, ROTATE_PROXY_ERRORS } from '@crawlee/utils';import { stringify } from 'csv-stringify/sync';-import ow, { ArgumentError } from 'ow';+import ow, { ArgumentError, type BasePredicate } from 'ow';import { getDomain } from 'tldts';import type { ReadonlyDeep, SetRequired } from 'type-fest';@@ -507,6 +511,21 @@ export interface BasicCrawlerOptions<*/id?: string;+ /**+ * Makes the storage writes performed while handling a request atomic with respect to the request+ * succeeding: they are recorded in a {@apilink StorageTransaction} spanning the whole request+ * lifecycle and only applied when the request handler succeeds, so a thrown handler leaves no partial+ * writes behind and a retry does not double-write. Reads within the handler see its own writes.+ *+ * `false` disables the mechanism entirely; an object overrides the per-storage-type+ * {@apilink StorageWritePolicy} (e.g. `{ requestQueue: 'deferred' }` for all-or-nothing enqueues).+ * {@apilink withDirectStorageAccess} is the per-call-site escape hatch; `useState()` is deliberately+ * *not* transactional.+ *+ * @default true+ */+ transactionalStorage?: boolean | Partial<StorageWritePolicy>;+/*** An array of HTTP response [Status Codes](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status) to be excluded from error consideration.* By default, status codes >= 500 trigger errors.@@ -788,6 +807,10 @@ export class BasicCrawler<protected readonly httpClient: BaseHttpClient;protected readonly retryOnBlocked: boolean;private respectRobotsTxtFile: boolean | { userAgent?: string };+ /** Whether `runInStorageTransaction()` opens a transaction at all. */+ private readonly transactionalStorageEnabled: boolean;+ /** The resolved per-storage-type write policy overrides forwarded to each request's transaction. */+ private readonly storageWritePolicy: Partial<StorageWritePolicy>;protected readonly onSkippedRequest?: SkippedRequestCallback;private _ownsEventManager = false;private loggedPerRun = new Set<string>();@@ -829,6 +852,12 @@ export class BasicCrawler<blockedStatusCodes: ow.optional.array.ofType(ow.number),retryOnBlocked: ow.optional.boolean,respectRobotsTxtFile: ow.optional.any(ow.boolean, ow.object),+ transactionalStorage: ow.optional.any(+ ow.boolean,+ ow.object.exactShape({+ requestQueue: ow.optional.string.oneOf(['deferred', 'writeThrough']),+ }),+ ) as BasePredicate<boolean | Partial<StorageWritePolicy> | undefined>,onSkippedRequest: ow.optional.function,httpClient: ow.optional.object,@@ -890,6 +919,7 @@ export class BasicCrawler<blockedStatusCodes: blockedStatusCodesInput,retryOnBlocked = false,respectRobotsTxtFile = false,+ transactionalStorage,onSkippedRequest,requestHandler,requestHandlerTimeoutSecs,@@ -993,6 +1023,13 @@ export class BasicCrawler<this.retryOnBlocked = retryOnBlocked;this.respectRobotsTxtFile = respectRobotsTxtFile;+ // The cast undoes ow's assertion signature, which mangles `boolean | object` unions.+ const transactionalStorageOption = transactionalStorage as+ | boolean+ | Partial<StorageWritePolicy>+ | undefined;+ this.transactionalStorageEnabled = transactionalStorageOption !== false;+ this.storageWritePolicy = typeof transactionalStorageOption === 'object' ? transactionalStorageOption : {};this.onSkippedRequest = onSkippedRequest;// allow at least 5min for internal timeouts@@ -1079,14 +1116,19 @@ export class BasicCrawler<const crawlingContext = { request } as { request: Request } & Partial<CrawlingContext>;try {- // Navigation, the navigation hooks and the request handler are timed individually, but the- // phases between them are not, so a request could still get stuck indefinitely. This is the- // catch-all for that - see `raceWithTimeout` for why it is a bare timer, not a timeout frame.- await this.withRequestTimeout(- crawlingContext,- this.basicContextPipeline- .chain(this.contextPipeline)- .call(crawlingContext, (ctx) => this.handleRequest(ctx, source, request)),+ // The transaction spans the whole pipeline call, covering the navigation hooks+ // and `extendContext` too; `handleRequest` drives its outcome explicitly.+ await this.runInStorageTransaction(+ async () =>+ // Navigation, the navigation hooks and the request handler are timed individually, but the+ // phases between them are not, so a request could still get stuck indefinitely. This is the+ // catch-all for that - see `raceWithTimeout` for why it is a bare timer, not a timeout frame.+ await this.withRequestTimeout(+ crawlingContext,+ this.basicContextPipeline+ .chain(this.contextPipeline)+ .call(crawlingContext, (ctx) => this.handleRequest(ctx, source, request)),+ ),);} catch (error) {// ContextPipelineInterruptedError means the request was intentionally skipped@@ -1818,22 +1860,26 @@ export class BasicCrawler<}protected async handleSkippedRequest(options: Parameters<SkippedRequestCallback>[0]): Promise<void> {- if (options.reason === 'limit') {- this.logOncePerRun(- 'maxRequestsPerCrawl',- 'The number of requests enqueued by the crawler reached the maxRequestsPerCrawl limit of ' +- `${this.maxRequestsPerCrawl} requests and no further requests will be added.`,- );- }+ // A skipped request is a *successful* outcome, but the interrupt still unwinds through the+ // transaction scope, which rolls back - so the skip bookkeeping must write directly.+ await withDirectStorageAccess(async () => {+ if (options.reason === 'limit') {+ this.logOncePerRun(+ 'maxRequestsPerCrawl',+ 'The number of requests enqueued by the crawler reached the maxRequestsPerCrawl limit of ' ++ `${this.maxRequestsPerCrawl} requests and no further requests will be added.`,+ );+ }- if (options.reason === 'depth') {- this.logOncePerRun(- 'maxCrawlDepth',- `The crawler reached the maxCrawlDepth limit of ${this.maxCrawlDepth} and no further requests will be enqueued.`,- );- }+ if (options.reason === 'depth') {+ this.logOncePerRun(+ 'maxCrawlDepth',+ `The crawler reached the maxCrawlDepth limit of ${this.maxCrawlDepth} and no further requests will be enqueued.`,+ );+ }- await this.onSkippedRequest?.(options);+ await this.onSkippedRequest?.(options);+ });}private logOncePerRun(key: string, message: string): void {@@ -2119,6 +2165,46 @@ export class BasicCrawler<);}+ /**+ * Runs `callback` inside a {@apilink StorageTransaction}, unless transactional storage is disabled.+ * Deliberately does **not** commit on return - `handleRequest` swallows request handler failures, so+ * a normal return says nothing about success. `handleRequest` owns the outcome.+ */+ private async runInStorageTransaction<T>(callback: () => Promise<T>): Promise<T> {+ if (!this.transactionalStorageEnabled) {+ return callback();+ }++ const transaction = createStorageTransaction({+ policy: this.storageWritePolicy,+ commitTimeoutMillis: this.internalTimeoutMillis,+ });++ let threw = true;+ try {+ const result = await transaction.run(callback);+ threw = false;+ return result;+ } finally {+ if (transaction.state === 'open') {+ // `handleRequest` commits or rolls back on every normal path, so an open transaction on+ // a normal return is a wiring bug; on a propagating throw (a pipeline-level failure) it is+ // expected. Either way, discard the unvalidated writes; only the former is worth flagging.+ if (!threw) {+ this.log.error(+ 'Internal error: a storage transaction was still open after the request pipeline ' ++ 'returned normally. Its writes are being discarded. Please report this.',+ );+ }++ transaction.rollback();+ }++ // Unconditional: `failed` is a terminal state that the branch above never reaches.+ transaction.dispose();+ }+ }+/*** Handles blocked request*/@@ -2260,12 +2346,19 @@ export class BasicCrawler<const statisticsId = request.id || request.uniqueKey;+ // Opened by `runInStorageTransaction`; absent when disabled or when the subclass opens its own.+ const transaction = currentStorageTransaction();+let isRequestLocked = true;try {request.state = RequestState.REQUEST_HANDLER;await this.runRequestHandler(crawlingContext);+ // Commit *before* marking the request as handled, so a commit failure fails the request and+ // it is retried. This also closes the transaction, so everything below passes through.+ await transaction?.commit();+await this.timeoutAndRetry(async () => requestSource.markRequestAsHandled(request!),this.internalTimeoutMillis,@@ -2281,6 +2374,10 @@ export class BasicCrawler<request.state = RequestState.DONE;crawlingContext.session.markGood();} catch (rawError) {+ // Roll back *before* any error handler runs - error handlers write to real storage precisely+ // because the transaction is already closed. A no-op when the commit above succeeded.+ transaction?.rollback();+const err = this.unwrapError(rawError);try {diff --git a/packages/browser-pool/src/browser-pool.ts b/packages/browser-pool/src/browser-pool.tsindex 4344ab7c6..5f283a651 100644--- a/packages/browser-pool/src/browser-pool.ts+++ b/packages/browser-pool/src/browser-pool.ts@@ -480,6 +480,9 @@ export class BrowserPool<// requests (https://github.com/apify/crawlee/issues/3670). Mirrors the// fix p-limit landed upstream in v5 (sindresorhus/p-limit#71); v5 is an// ESM-only rewrite, so we can't bump it in Crawlee v3.+ // Besides the cancelTask leak, the wrapper also keeps the per-request *storage transaction*+ // ALS-scoped: without it, a queued callback would resume in the previous request's async+ // context and run request B's storage writes inside request A's transaction.// TODO(crawlee@v4): bump p-limit to v5 and drop this AsyncResource.bind wrapper.// Limiter is necessary - https://github.com/apify/crawlee/issues/1126return this.limiter(diff --git a/packages/core/src/crawlers/crawler_commons.ts b/packages/core/src/crawlers/crawler_commons.tsindex 80bb8cd4c..8edab7fa6 100644--- a/packages/core/src/crawlers/crawler_commons.ts+++ b/packages/core/src/crawlers/crawler_commons.ts@@ -1,13 +1,12 @@import type { Dictionary, HttpRequestOptions, ISession, ProxyInfo, SendRequestOptions } from '@crawlee/types';import type { ReadonlyDeep, SetRequired } from 'type-fest';-import type { Configuration } from '../configuration.js';import type { EnqueueLinksOptions } from '../enqueue_links/enqueue_links.js';import type { CrawleeLogger } from '../log.js';import type { Request, RequestOptions, Source } from '../request.js';import type { StorageIdentifier } from '../storages/storage_instance_manager.js';import type { Dataset } from '../storages/dataset.js';-import { KeyValueStore, type RecordOptions } from '../storages/key_value_store.js';+import type { KeyValueStore } from '../storages/key_value_store.js';import type { RequestQueueOperationOptions } from '../storages/request_queue.js';/** @internal */@@ -229,6 +228,11 @@ export interface CrawlingContext<UserData extends Dictionary = Dictionary> exten/*** Register a function to be called at the very end of the request handling process. This is useful for resources that should be accessible to error handlers, for instance.+ *+ * The callback runs *outside* the request's storage transaction, so storage writes made here are+ * applied immediately and are **not** rolled back when the request fails. In+ * {@apilink AdaptivePlaywrightCrawler} it also runs once per request handler attempt, so a write+ * here can land more than once for a single request. Push results from the request handler itself.*/registerDeferredCleanup(cleanup: () => Promise<unknown>): void;@@ -252,143 +256,3 @@ export interface CrawlingContext<UserData extends Dictionary = Dictionary> exten*/extendTimeout(secs: number): void;}--/**- * A partial implementation of {@apilink RestrictedCrawlingContext} that stores parameters of calls to context methods for later inspection.- *- * @experimental- */-export class RequestHandlerResult {- private _keyValueStoreChanges: Record<string, Record<string, { changedValue: unknown; options?: RecordOptions }>> =- {};-- private pushDataCalls: Parameters<RestrictedCrawlingContext['pushData']>[] = [];-- private addRequestsCalls: Parameters<RestrictedCrawlingContext['addRequests']>[] = [];-- constructor(- private configuration: Configuration,- private crawleeStateKey: string,- ) {}-- /**- * A record of calls to {@apilink RestrictedCrawlingContext.pushData}, {@apilink RestrictedCrawlingContext.addRequests}, {@apilink RestrictedCrawlingContext.enqueueLinks} made by a request handler.- */- get calls(): ReadonlyDeep<{- pushData: Parameters<RestrictedCrawlingContext['pushData']>[];- addRequests: Parameters<RestrictedCrawlingContext['addRequests']>[];- }> {- return {- pushData: this.pushDataCalls,- addRequests: this.addRequestsCalls,- };- }-- /**- * A record of changes made to key-value stores by a request handler.- */- get keyValueStoreChanges(): ReadonlyDeep<- Record<string, Record<string, { changedValue: unknown; options?: RecordOptions }>>- > {- return this._keyValueStoreChanges;- }-- /**- * Items added to datasets by a request handler.- */- get datasetItems(): ReadonlyDeep<{ item: Dictionary; datasetIdentifier?: string | StorageIdentifier }[]> {- return this.pushDataCalls.flatMap(([data, datasetIdentifier]) =>- (Array.isArray(data) ? data : [data]).map((item) => ({ item, datasetIdentifier })),- );- }-- /**- * URLs enqueued to the request queue by a request handler, either via {@apilink RestrictedCrawlingContext.addRequests} or {@apilink RestrictedCrawlingContext.enqueueLinks}- */- get enqueuedUrls(): ReadonlyDeep<{ url: string; label?: string }[]> {- const result: { url: string; label?: string }[] = [];-- for (const [requests] of this.addRequestsCalls) {- for (const request of requests) {- if (- typeof request === 'object' &&- (!('requestsFromUrl' in request) || request.requestsFromUrl !== undefined) &&- request.url !== undefined- ) {- result.push({ url: request.url, label: request.label });- } else if (typeof request === 'string') {- result.push({ url: request });- }- }- }-- return result;- }-- /**- * URL lists enqueued to the request queue by a request handler via {@apilink RestrictedCrawlingContext.addRequests} using the `requestsFromUrl` option.- */- get enqueuedUrlLists(): ReadonlyDeep<{ listUrl: string; label?: string }[]> {- const result: { listUrl: string; label?: string }[] = [];-- for (const [requests] of this.addRequestsCalls) {- for (const request of requests) {- if (- typeof request === 'object' &&- 'requestsFromUrl' in request &&- request.requestsFromUrl !== undefined- ) {- result.push({ listUrl: request.requestsFromUrl, label: request.label });- }- }- }-- return result;- }-- pushData: RestrictedCrawlingContext['pushData'] = async (data, datasetIdOrName) => {- this.pushDataCalls.push([data, datasetIdOrName]);- };-- addRequests: RestrictedCrawlingContext['addRequests'] = async (requests, options = {}) => {- this.addRequestsCalls.push([requests, options]);- };-- useState: RestrictedCrawlingContext['useState'] = async (defaultValue) => {- const store = await this.getKeyValueStore(undefined);- return await store.getAutoSavedValue(this.crawleeStateKey, defaultValue);- };-- getKeyValueStore: RestrictedCrawlingContext['getKeyValueStore'] = async (identifier) => {- const store = await KeyValueStore.open(identifier, { configuration: this.configuration });- const storeId = store.id;-- return {- id: storeId ?? this.configuration.defaultKeyValueStoreId,- name: store.name,- getValue: async (key) => this.getKeyValueStoreChangedValue(storeId, key) ?? (await store.getValue(key)),- setValue: async (key, value, options) => {- this.setKeyValueStoreChangedValue(storeId, key, value, options);- },- getAutoSavedValue: store.getAutoSavedValue.bind(store),- getPublicUrl: store.getPublicUrl.bind(store),- };- };-- private getKeyValueStoreChangedValue = (storeKey: string | undefined, key: string) => {- const id = storeKey ?? this.configuration.defaultKeyValueStoreId;- this._keyValueStoreChanges[id] ??= {};- return this.keyValueStoreChanges[id][key]?.changedValue ?? null;- };-- private setKeyValueStoreChangedValue = (- storeKey: string | undefined,- key: string,- changedValue: unknown,- options?: RecordOptions,- ) => {- const id = storeKey ?? this.configuration.defaultKeyValueStoreId;- this._keyValueStoreChanges[id] ??= {};- this._keyValueStoreChanges[id][key] = { changedValue, options };- };-}diff --git a/packages/core/src/crawlers/index.ts b/packages/core/src/crawlers/index.tsindex fc8ce2593..9a1018454 100644--- a/packages/core/src/crawlers/index.ts+++ b/packages/core/src/crawlers/index.ts@@ -1,5 +1,5 @@export * from './context_pipeline.js';-export * from './crawler_commons.js';+export type * from './crawler_commons.js';export * from './statistics.js';export * from './error_tracker.js';export * from './error_snapshotter.js';diff --git a/packages/core/src/storages/access_checking.ts b/packages/core/src/storages/access_checking.tsdeleted file mode 100644index 016dee6fb..000000000--- a/packages/core/src/storages/access_checking.ts+++ /dev/null@@ -1,23 +0,0 @@-import { AsyncLocalStorage } from 'node:async_hooks';--import type { Awaitable } from '@crawlee/types';-import { tryCancel } from '@apify/timeout';--const storage = new AsyncLocalStorage<{ checkFunction: () => void }>();--/**- * Invoke a storage access checker function defined using {@link withCheckedStorageAccess} higher up in the call stack.- */-export const checkStorageAccess = () => {- tryCancel();- return storage.getStore()?.checkFunction();-};--/**- * Define a storage access checker function that should be used by calls to {@link checkStorageAccess} in the callbacks.- *- * @param checkFunction The check function that should be invoked by {@link checkStorageAccess} calls- * @param callback The code that should be invoked with the `checkFunction` setting- */-export const withCheckedStorageAccess = async <T>(checkFunction: () => void, callback: () => Awaitable<T>) =>- storage.run({ checkFunction }, callback);diff --git a/packages/core/src/storages/dataset.ts b/packages/core/src/storages/dataset.tsindex 5c40668db..4f71cb166 100644--- a/packages/core/src/storages/dataset.ts+++ b/packages/core/src/storages/dataset.ts@@ -2,10 +2,13 @@ import type { Awaitable, DatasetBackend, DatasetInfo, Dictionary, PaginatedListimport { stringify } from 'csv-stringify/sync';import ow from 'ow';+import { tryCancel } from '@apify/timeout';+import { Configuration } from '../configuration.js';import type { CrawleeLogger } from '../log.js';import { serviceLocator } from '../service_locator.js';-import { checkStorageAccess } from './access_checking.js';+import type { DatasetJournalEntry, JournalEntry } from './transaction.js';+import { activeStorageTransaction, rejectOperationInTransaction, snapshotValue } from './transaction.js';import { KeyValueStore } from './key_value_store.js';import type { DatasetStats } from './storage_stats.js';import { StorageStatsTracker } from './storage_stats.js';@@ -227,7 +230,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {* The objects must be serializable to JSON.*/async pushData(data: Data | Data[]): Promise<void> {- checkStorageAccess();+ const transaction = activeStorageTransaction();ow(data, 'data', ow.object);@@ -237,6 +240,18 @@ export class Dataset<Data extends Dictionary = Dictionary> {assertJsonSerializable(items[i], i);}+ if (transaction) {+ // One snapshot serves both the reads and the commit replay, so the two cannot disagree.+ transaction.recordJournalEntry({+ type: 'dataset',+ participant: this,+ storageId: this.id,+ items: snapshotValue(items),+ recordedAt: new Date(),+ });+ return;+ }+this.statsTracker.add('writeCount');await this.backend.pushData(items);}@@ -245,11 +260,8 @@ export class Dataset<Data extends Dictionary = Dictionary> {* Returns {@apilink DatasetContent} object holding the items in the dataset based on the provided parameters.*/async getData(options: DatasetDataOptions = {}): Promise<DatasetContent<Data>> {- checkStorageAccess();-try {- this.statsTracker.add('readCount');- return await this.backend.getData(options);+ return await this.readPage(options);} catch (e) {const error = e as Error;if (error.message.includes('Cannot create a string longer than')) {@@ -261,12 +273,111 @@ export class Dataset<Data extends Dictionary = Dictionary> {}}+ /**+ * The single transaction-aware page read all dataset read paths go through — both `getData()` and+ * the private `fetchPages()`. Returns the real page concatenated with the current transaction's+ * buffered items, with `offset` / `limit` / `desc` windowing applied across the concatenation.+ */+ private async readPage(options: DatasetDataOptions): Promise<DatasetContent<Data>> {+ const buffered = this.bufferedJournalEntries()?.flatMap((entry) => entry.items as Data[]);++ // Every branch below hits the backend exactly once.+ this.statsTracker.add('readCount');++ if (!buffered?.length) {+ return this.backend.getData(options);+ }++ const { offset = 0, limit, desc = false } = options;++ if (!desc) {+ const realPage = await this.backend.getData(options);++ // Buffered items sit past `realPage.total`, so the window bounds must come from that - not+ // from the page's shortfall, which `skipEmpty` produces without exhausting the real items.+ const bufferedStart = Math.max(0, offset - realPage.total);+ const bufferedEnd = limit === undefined ? buffered.length : Math.max(0, offset + limit - realPage.total);+ const items = [...realPage.items, ...buffered.slice(bufferedStart, bufferedEnd)];++ return {+ items,+ total: realPage.total + buffered.length,+ offset,+ // A caller that passed no limit wants everything, so the backend-reported `limit` is+ // passed through - backends are free to report a page size or a sentinel there.+ limit: limit ?? realPage.limit,+ count: items.length,+ desc,+ };+ }++ // Descending order: the buffered items are the newest, so they come first, reversed.+ const reversedBuffer = [...buffered].reverse();+ const fromBuffer =+ limit === undefined ? reversedBuffer.slice(offset) : reversedBuffer.slice(offset, offset + limit);+ const needed = limit === undefined ? Infinity : limit - fromBuffer.length;++ if (needed <= 0) {+ // The whole window is served from the buffer; only the real total is missing.+ const { itemCount } = await this.backend.getMetadata();+ return {+ items: fromBuffer,+ total: itemCount + buffered.length,+ offset,+ limit: limit!,+ count: fromBuffer.length,+ desc,+ };+ }++ const realPage = await this.backend.getData({+ ...options,+ offset: Math.max(0, offset - buffered.length),+ ...(limit === undefined ? {} : { limit: needed }),+ });++ return {+ items: [...fromBuffer, ...realPage.items],+ total: realPage.total + buffered.length,+ offset,+ limit: limit ?? realPage.limit,+ count: fromBuffer.length + realPage.items.length,+ desc,+ };+ }++ /** The active transaction's buffered writes to this dataset, derived from its journal. */+ private bufferedJournalEntries(): DatasetJournalEntry[] | undefined {+ const transaction = activeStorageTransaction();+ return transaction?.journal.filter(+ (entry): entry is DatasetJournalEntry => entry.type === 'dataset' && entry.participant === this,+ );+ }++ /** @internal */+ async commitJournalEntries(entries: JournalEntry[]): Promise<void> {+ const items: Data[] = [];++ for (const entry of entries) {+ if (entry.type === 'dataset') {+ items.push(...(entry.items as Data[]));+ }+ }++ // One backend call with all journaled items, in order - as close to atomic as the backend allows.+ // Straight to the backend: the items were validated and snapshotted at write time.+ if (items.length > 0) {+ this.statsTracker.add('writeCount');+ await this.backend.pushData(items);+ }+ }+/*** Returns all the data from the dataset. This will iterate through the whole dataset* via the `listItems()` client method, which gives you only paginated results.*/async export(options: DatasetExportOptions = {}): Promise<Data[]> {- checkStorageAccess();+ tryCancel();const items: Data[] = [];@@ -344,7 +455,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {* @param [options] An optional options object where you can provide the dataset and target KVS name.*/static async exportToJSON(key: string, options?: DatasetExportToOptions) {- checkStorageAccess();+ tryCancel();const dataset = await this.open(options?.fromDataset);await dataset.exportToJSON(key, options);@@ -357,7 +468,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {* @param [options] An optional options object where you can provide the dataset and target KVS name.*/static async exportToCSV(key: string, options?: DatasetExportToOptions) {- checkStorageAccess();+ tryCancel();const dataset = await this.open(options?.fromDataset);await dataset.exportToCSV(key, options);@@ -381,9 +492,19 @@ export class Dataset<Data extends Dictionary = Dictionary> {* @throws If the underlying storage no longer exists (e.g. it was deleted externally).*/async getInfo(): Promise<DatasetInfo> {- checkStorageAccess();+ const buffered = this.bufferedJournalEntries();+ const metadata = await this.backend.getMetadata();++ if (buffered?.length) {+ const lastWriteAt = buffered[buffered.length - 1].recordedAt;+ return {+ ...metadata,+ itemCount: metadata.itemCount + buffered.reduce((sum, entry) => sum + entry.items.length, 0),+ modifiedAt: metadata.modifiedAt > lastWriteAt ? metadata.modifiedAt : lastWriteAt,+ };+ }- return this.backend.getMetadata();+ return metadata;}/**@@ -407,7 +528,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {* @default 0*/async forEach(iteratee: DatasetConsumer<Data>, options: DatasetIteratorOptions = {}, index = 0): Promise<void> {- checkStorageAccess();+ tryCancel();if (!options.offset) options.offset = 0;if (options.format && options.format !== 'json')@@ -437,7 +558,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {* @param [options] All `map()` parameters.*/async map<R>(iteratee: DatasetMapper<Data, R>, options: DatasetIteratorOptions = {}): Promise<R[]> {- checkStorageAccess();+ tryCancel();const result: R[] = [];@@ -513,7 +634,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {memo?: T,options: DatasetIteratorOptions = {},): Promise<T | undefined> {- checkStorageAccess();+ tryCancel();let currentMemo: T | undefined = memo;@@ -553,8 +674,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {const fetchLimit = totalLimit !== undefined ? Math.min(pageSize, totalLimit - yielded) : pageSize;if (fetchLimit <= 0) break;- this.statsTracker.add('readCount');- const page = await this.backend.getData({ ...options, offset, limit: fetchLimit });+ const page = await this.readPage({ ...options, offset, limit: fetchLimit });yield page;yielded += page.items.length;@@ -587,7 +707,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {* @param options Options for the iteration.*/values(options: DatasetIteratorOptions = {}): AsyncIterable<Data> & Promise<Data[]> {- checkStorageAccess();+ tryCancel();return createDualIterable({createPages: () => this.fetchPages(options),@@ -619,7 +739,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {* @param options Options for the iteration.*/entries(options: DatasetIteratorOptions = {}): AsyncIterable<[number, Data]> & Promise<[number, Data][]> {- checkStorageAccess();+ tryCancel();return createDualIterable({createPages: () => this.fetchEntryPages(options),@@ -648,7 +768,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {* depending on the mode of operation.*/async drop(): Promise<void> {- checkStorageAccess();+ rejectOperationInTransaction('Dataset.drop()');await this.backend.drop();serviceLocator.getStorageInstanceManager().removeFromCache(this);@@ -673,7 +793,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {identifier?: string | StorageIdentifier | null,options: StorageOpenOptions = {},): Promise<Dataset<Data>> {- checkStorageAccess();+ tryCancel();ow(options,diff --git a/packages/core/src/storages/index.ts b/packages/core/src/storages/index.tsindex 7d596e062..af38af425 100644--- a/packages/core/src/storages/index.ts+++ b/packages/core/src/storages/index.ts@@ -8,6 +8,6 @@ export * from './request_queue.js';export * from './storage_instance_manager.js';export * from './storage_stats.js';export * from './utils.js';-export * from './access_checking.js';+export * from './transaction.js';export * from './sitemap_request_loader.js';export * from './request_manager_tandem.js';diff --git a/packages/core/src/storages/key_value_store.ts b/packages/core/src/storages/key_value_store.tsindex 13369d624..286188fe1 100644--- a/packages/core/src/storages/key_value_store.ts+++ b/packages/core/src/storages/key_value_store.ts@@ -8,10 +8,18 @@ import type {import ow, { ArgumentError } from 'ow';import { KEY_VALUE_STORE_KEY_REGEX } from '@apify/consts';+import { tryCancel } from '@apify/timeout';import { Configuration } from '../configuration.js';import { serviceLocator } from '../service_locator.js';-import { checkStorageAccess } from './access_checking.js';+import type { JournalEntry, KeyValueStoreJournalEntry } from './transaction.js';+import {+ activeStorageTransaction,+ operationRejectedInTransaction,+ rejectOperationInTransaction,+ snapshotValue,+ withDirectStorageAccess,+} from './transaction.js';import { parseValue, serializeValue } from './key_value_store_codec.js';import type { KeyValueStoreStats } from './storage_stats.js';import { StorageStatsTracker } from './storage_stats.js';@@ -213,11 +221,10 @@ export class KeyValueStore {* on the MIME content type of the record, or `null` if the key is missing from the store.*/async getValue<T = unknown>(key: string, defaultValue?: T): Promise<T | null> {- checkStorageAccess();+ tryCancel();ow(key, ow.string.nonEmpty);- this.statsTracker.add('readCount');- const record = await this.backend.getValue(key);+ const record = await this.readRecord(key);// A missing record falls back to the default; a record that parses to a falsy value (including// a stored literal `null`) is returned verbatim, so callers can tell "stored null" from "absent".@@ -229,6 +236,62 @@ export class KeyValueStore {return parseValue(record.value, record.contentType ?? null) as T;}+ /**+ * The active transaction's last buffered write per key for this store, derived from its journal.+ * An entry with a `null` value is a tombstone (an in-transaction deletion).+ */+ private bufferedJournalEntries(): Map<string, KeyValueStoreJournalEntry> | undefined {+ const transaction = activeStorageTransaction();+ if (!transaction) return undefined;++ const lastWritePerKey = new Map<string, KeyValueStoreJournalEntry>();++ for (const entry of transaction.journal) {+ if (entry.type === 'keyValueStore' && entry.participant === this) {+ lastWritePerKey.set(entry.key, entry);+ }+ }++ return lastWritePerKey;+ }++ /**+ * The single transaction-aware record read shared by `getValue`, `getRecord`, `recordExists` and the+ * listing paths: buffered key → serialized through the standard codec (same fidelity as a real+ * round-trip); tombstoned key → `null`; otherwise the backend.+ *+ * The per-key buffered lookup requires the whole journal to be reduced to a last-write-per-key map,+ * which is O(journal). Single-record callers let it default (rebuilt per call); the listing paths,+ * which read many keys, pass a map built once so the read stays O(1) per key instead of O(journal).+ */+ private async readRecord(+ key: string,+ buffered = this.bufferedJournalEntries(),+ ): Promise<{ value: Buffer | ArrayBuffer; contentType: string | null } | null> {+ const entry = buffered?.get(key);++ if (entry) {+ if (entry.value === null) {+ return null;+ }++ const serialized = serializeValue(entry.value, entry.options?.contentType);+ return {+ value: normalizeSerializedValue(serialized.value),+ contentType: serialized.contentType ?? null,+ };+ }++ this.statsTracker.add('readCount');+ const record = await this.backend.getValue(key);+ if (!record) return null;++ return {+ value: record.value,+ contentType: record.contentType ?? null,+ };+ }+/*** Reads a record from the key-value store without parsing the value.*@@ -256,17 +319,10 @@ export class KeyValueStore {* of the following characters: `a`-`z`, `A`-`Z`, `0`-`9` and `!-_.'()`*/async getRecord(key: string): Promise<KeyValueStoreRawRecord | null> {- checkStorageAccess();+ tryCancel();ow(key, ow.string.nonEmpty);- this.statsTracker.add('readCount');- const record = await this.backend.getValue(key);- if (!record) return null;-- return {- value: record.value,- contentType: record.contentType ?? null,- };+ return this.readRecord(key);}/**@@ -276,20 +332,28 @@ export class KeyValueStore {* @returns `true` if the record exists, `false` if it does not.*/async recordExists(key: string): Promise<boolean> {- checkStorageAccess();+ tryCancel();ow(key, ow.string.nonEmpty);++ const entry = this.bufferedJournalEntries()?.get(key);+ if (entry) {+ return entry.value !== null;+ }+return this.backend.recordExists(key);}async getAutoSavedValue<T extends Dictionary = Dictionary>(key: string, defaultValue = {} as T): Promise<T> {- checkStorageAccess();+ tryCancel();if (this.cache.has(key)) {return this.cache.get(key) as T;}- const value = await this.getValue<T>(key, defaultValue);+ // Auto-saved state is deliberately *not* transactional. The direct read bypasses any active+ // transaction - a buffered value seeded into this shared cache would survive a rollback forever.+ const value = await withDirectStorageAccess(async () => this.getValue<T>(key, defaultValue));// The await above could have run in parallel with another call to this function. If the other call finished more quickly,// the value will in cache at this point, and returning the new fetched value would introduce two different instances of@@ -331,11 +395,15 @@ export class KeyValueStore {options: KeyValueStoreIteratorOptions,mapRecord: (key: string, value: unknown) => T,): AsyncGenerator<T[]> {- for await (const page of this.fetchKeyPages(options)) {+ // Reduce the journal once for the whole iteration, not once per key inside `readRecord`.+ const buffered = this.bufferedJournalEntries();++ for await (const page of this.fetchKeyPages(options, buffered)) {const results: T[] = [];for (const item of page) {- this.statsTracker.add('readCount');- const record = await this.backend.getValue(item.key);+ // The shared transaction-aware read, so a key that exists only in the transaction resolves+ // here instead of being dropped (`values()` would disagree with `keys()` on length).+ const record = await this.readRecord(item.key, buffered);if (record) {const parsed = parseValue(record.value, record.contentType ?? null);results.push(mapRecord(item.key, parsed));@@ -347,8 +415,31 @@ export class KeyValueStore {private async *fetchKeyPages(options: KeyValueStoreIteratorOptions,+ buffered = this.bufferedJournalEntries(),limit = KVS_KEYS_DEFAULT_LIMIT,): AsyncGenerator<KeyValueStoreItemData[]> {+ // Buffered keys are emitted first, then the real pages with any buffered (or tombstoned) key+ // skipped - a merge-join is not an option, since `listKeys` promises no sort order.+ const shadowedKeys = new Set<string>();++ if (buffered) {+ const bufferedItems: KeyValueStoreItemData[] = [];++ for (const [key, entry] of buffered) {+ shadowedKeys.add(key);++ if (entry.value === null) continue;+ if (options.prefix !== undefined && !key.startsWith(options.prefix)) continue;++ bufferedItems.push(bufferedKeyItemData(key, entry));+ }++ if (bufferedItems.length > 0) {+ bufferedItems.sort((a, b) => (a.key < b.key ? -1 : 1));+ yield bufferedItems;+ }+ }+let exclusiveStartKey: string | undefined;while (true) {@@ -358,8 +449,9 @@ export class KeyValueStore {exclusiveStartKey,limit,});- yield items;+ yield shadowedKeys.size > 0 ? items.filter((item) => !shadowedKeys.has(item.key)) : items;if (!isTruncated) break;+ // Paginate from the raw backend cursor - it may reject a key it did not hand out.exclusiveStartKey = nextExclusiveStartKey;}}@@ -408,7 +500,7 @@ export class KeyValueStore {* @param [options] Record options.*/async setValue<T>(key: string, value: T | null, options: RecordOptions = {}): Promise<void> {- checkStorageAccess();+ const transaction = activeStorageTransaction();ow(key, 'key', ow.string.nonEmpty);ow(@@ -434,6 +526,38 @@ export class KeyValueStore {// Make copy of options, don't update what user passed.const optionsCopy = { ...options };+ // The whole transaction branch sits *above* the auto-saved cache update below, so a buffered+ // write touches nothing outside the journal. That cache is shared, process-lifetime frontend+ // state, so mutating it here would survive a rollback and later be persisted by `persistState`.+ // The commit replay re-enters this method with no active transaction and updates it then.+ if (transaction) {+ if (isStream(value)) {+ // A stream cannot serve both a read-your-own-writes read and the commit replay. The+ // transaction is known-active here, so throw directly rather than via the conditional guard.+ throw operationRejectedInTransaction(+ `KeyValueStore.setValue() with a stream value (key "${key}")`,+ 'a stream can only be consumed once, so it cannot be buffered until commit.',+ );+ }++ // Validation only, result discarded: the journal snapshot (`structuredClone`) accepts values+ // JSON cannot, which would otherwise only throw at a later read or at commit.+ if (value !== null) {+ serializeValue(value, optionsCopy.contentType);+ }++ // One snapshot serves both the reads and the commit replay; `null` is a tombstone.+ transaction.recordJournalEntry({+ type: 'keyValueStore',+ participant: this,+ storageId: this.id,+ key,+ value: value === null ? null : snapshotValue(value),+ options: optionsCopy,+ });+ return;+ }+// If we try to set the value of a cached state to a different reference, we need to update the cache accordingly.const cachedValue = this.cache.get(key);@@ -467,12 +591,28 @@ export class KeyValueStore {});}+ /** @internal */+ async commitJournalEntries(entries: JournalEntry[]): Promise<void> {+ // One `setValue` per key, last write wins - idempotent under retry.+ const lastWritePerKey = new Map<string, { value: unknown; options?: RecordOptions }>();++ for (const entry of entries) {+ if (entry.type === 'keyValueStore') {+ lastWritePerKey.set(entry.key, { value: entry.value, options: entry.options });+ }+ }++ for (const [key, { value, options }] of lastWritePerKey) {+ await this.setValue(key, value, options);+ }+ }+/*** Removes the key-value store either from the Apify cloud storage or from the local directory,* depending on the mode of operation.*/async drop(): Promise<void> {- checkStorageAccess();+ rejectOperationInTransaction('KeyValueStore.drop()');await this.backend.drop();serviceLocator.getStorageInstanceManager().removeFromCache(this);@@ -480,7 +620,7 @@ export class KeyValueStore {/** @internal */clearCache(): void {- checkStorageAccess();+ rejectOperationInTransaction('KeyValueStore.clearCache()');this.cache.clear();}@@ -507,7 +647,7 @@ export class KeyValueStore {* @param [options] All `forEachKey()` parameters.*/async forEachKey(iteratee: KeyConsumer, options: KeyValueStoreIteratorOptions = {}): Promise<void> {- checkStorageAccess();+ tryCancel();ow(iteratee, ow.function);ow(@@ -550,7 +690,7 @@ export class KeyValueStore {* @param options Options for the iteration.*/keys(options: KeyValueStoreIteratorOptions = {}): AsyncIterable<string> & Promise<string[]> {- checkStorageAccess();+ tryCancel();return createDualIterable({createPages: () => this.fetchKeyPages(options),@@ -582,7 +722,7 @@ export class KeyValueStore {* @param options Options for the iteration.*/values<T = unknown>(options: KeyValueStoreIteratorOptions = {}): AsyncIterable<T> & Promise<T[]> {- checkStorageAccess();+ tryCancel();return createDualIterable({createPages: () => this.fetchKeyValuePages<T>(options, (_key, value) => value as T),@@ -616,7 +756,7 @@ export class KeyValueStore {entries<T = unknown>(options: KeyValueStoreIteratorOptions = {},): AsyncIterable<[string, T]> & Promise<[string, T][]> {- checkStorageAccess();+ tryCancel();return createDualIterable({createPages: () => this.fetchKeyValuePages<[string, T]>(options, (key, value) => [key, value as T]),@@ -670,7 +810,7 @@ export class KeyValueStore {identifier?: string | StorageIdentifier | null,options: StorageOpenOptions = {},): Promise<KeyValueStore> {- checkStorageAccess();+ tryCancel();ow(options,@@ -880,6 +1020,30 @@ export class KeyValueStore {}}+/** Normalizes a codec-serialized value into the `Buffer | ArrayBuffer` shape raw record reads promise. */+function normalizeSerializedValue(value: ReturnType<typeof serializeValue>['value']): Buffer | ArrayBuffer {+ if (typeof value === 'string') {+ return Buffer.from(value);+ }++ if (ArrayBuffer.isView(value)) {+ return Buffer.isBuffer(value) ? value : Buffer.from(value.buffer, value.byteOffset, value.byteLength);+ }++ return value as Buffer | ArrayBuffer;+}++/** Computes the key listing item (serialized byte size and content type) of a buffered entry. */+function bufferedKeyItemData(key: string, entry: { value: unknown; options?: RecordOptions }): KeyValueStoreItemData {+ const serialized = serializeValue(entry.value, entry.options?.contentType);++ return {+ key,+ size: normalizeSerializedValue(serialized.value).byteLength,+ contentType: serialized.contentType,+ };+}+/*** User-function used in the {@apilink KeyValueStore.forEachKey} method.*/diff --git a/packages/core/src/storages/request_queue.ts b/packages/core/src/storages/request_queue.tsindex a95bfc488..e01b5ab9b 100644--- a/packages/core/src/storages/request_queue.ts+++ b/packages/core/src/storages/request_queue.ts@@ -15,6 +15,7 @@ import ow from 'ow';import type { ReadonlyDeep } from 'type-fest';import { LruCache } from '@apify/datastructures';+import { tryCancel } from '@apify/timeout';import { Configuration } from '../configuration.js';import { getObjectType } from '../debug.js';@@ -26,7 +27,8 @@ import type { IProxyConfiguration } from '../proxy_configuration.js';import type { InternalSource, RequestOptions, Source } from '../request.js';import { Request } from '../request.js';import { serviceLocator } from '../service_locator.js';-import { checkStorageAccess } from './access_checking.js';+import type { JournalEntry, StorageTransaction } from './transaction.js';+import { activeStorageTransaction, rejectOperationInTransaction, withDirectStorageAccess } from './transaction.js';import type { IRequestManager, RequestsLike } from './request_manager.js';import type { RequestQueueStats } from './storage_stats.js';import { StorageStatsTracker } from './storage_stats.js';@@ -179,7 +181,7 @@ export class RequestQueue implements IStorage, IRequestManager {requestLike: Source,options: RequestQueueOperationOptions = {},): Promise<RequestQueueOperationInfo> {- checkStorageAccess();+ const transaction = activeStorageTransaction();ow(requestLike, ow.object);ow(@@ -208,11 +210,16 @@ export class RequestQueue implements IStorage, IRequestManager {const request = requestLike instanceof Request ? requestLike : new Request(requestLike);+ if (transaction?.policy.requestQueue === 'deferred') {+ return this.addRequestDeferred(transaction, request, forefront);+ }+const cacheKey = getRequestId(request.uniqueKey);const cachedInfo = this.requestCache.get(cacheKey);if (cachedInfo) {request.id = cachedInfo.id;+ this.recordRequestJournalEntry(transaction, [request], forefront, true);return {wasAlreadyPresent: true,// We may assume that if request is in local cache then also the information if the@@ -226,6 +233,7 @@ export class RequestQueue implements IStorage, IRequestManager {this.statsTracker.add('writeCount');const { processedRequests } = await this.backend.addBatchOfRequests([request], { forefront });+ this.recordRequestJournalEntry(transaction, [request], forefront, true);const queueOperationInfo = {...processedRequests[0],uniqueKey: request.uniqueKey,@@ -238,6 +246,182 @@ export class RequestQueue implements IStorage, IRequestManager {return queueOperationInfo;}+ /**+ * Journals an addition for introspection only; these entries are never replayed. A no-op unless the+ * transaction is open, so detached and outliving writers stay out of the journal.+ */+ private recordRequestJournalEntry(+ transaction: StorageTransaction | undefined,+ requests: Request[],+ forefront: boolean,+ writeThrough: boolean,+ ): void {+ if (!transaction?.isActive || requests.length === 0) return;++ transaction.recordJournalEntry({+ type: 'requestQueue',+ participant: this,+ requests: requests.map((request) => ({+ url: request.url,+ uniqueKey: request.uniqueKey,+ label: request.label,+ })),+ forefront,+ writeThrough,+ });+ }++ /**+ * The requests buffered by the given transaction for this queue, keyed by `uniqueKey` — a dedup+ * index derived from the transaction journal.+ */+ private bufferedRequests(transaction: StorageTransaction): Map<string, Dictionary> {+ const buffered = new Map<string, Dictionary>();++ // Only `deferred` records snapshots, so scanning the journal under `writeThrough` never finds any.+ if (transaction.policy.requestQueue !== 'deferred') return buffered;++ for (const entry of transaction.journal) {+ if (entry.type !== 'requestQueue' || entry.participant !== this) continue;+ for (const request of entry.requests) {+ if (request.snapshot !== undefined) buffered.set(request.uniqueKey, request.snapshot);+ }+ }++ return buffered;+ }++ /**+ * Adds a request under the `deferred` policy: journaled now, really added by the commit replay.+ * A new request's `requestId` is the local `uniqueKey` hash and is **provisional** — never write it+ * to `request.id` or the dedup caches. Dedup is cheapest-first: buffer, caches, then a backend probe.+ */+ private async addRequestDeferred(+ transaction: StorageTransaction,+ request: Request,+ forefront: boolean,+ buffered = this.bufferedRequests(transaction),+ ): Promise<RequestQueueOperationInfo> {+ // This transaction's own buffered adds; the shared caches never see them (provisional ids).+ if (buffered.has(request.uniqueKey)) {+ this.recordRequestJournalEntry(transaction, [request], forefront, false);+ return {+ wasAlreadyPresent: true,+ wasAlreadyHandled: false,+ requestId: getRequestId(request.uniqueKey),+ uniqueKey: request.uniqueKey,+ forefront,+ };+ }++ // The caches hold real backend ids. Only *writing* provisional ids to them would be wrong;+ // reading saves a probe. Same lookup as the write-through path.+ const cacheKey = getRequestId(request.uniqueKey);+ const cachedInfo = this.requestCache.get(cacheKey);+ const knownRequestId = cachedInfo?.id ?? this.requestSeenCache.get(cacheKey);++ if (knownRequestId) {+ this.recordRequestJournalEntry(transaction, [request], forefront, false);+ return {+ wasAlreadyPresent: true,+ // The dedup cache doesn't track the handled state; only the full record does.+ wasAlreadyHandled: cachedInfo?.isHandled ?? false,+ requestId: knownRequestId,+ uniqueKey: request.uniqueKey,+ forefront,+ };+ }++ // The caches are bounded, so a miss is not proof of absence - probe for an accurate answer.+ const existing = await this.backend.getRequest(request.uniqueKey);++ if (existing) {+ this.recordRequestJournalEntry(transaction, [request], forefront, false);+ return {+ wasAlreadyPresent: true,+ wasAlreadyHandled: existing.handledAt != null,+ requestId: existing.id,+ uniqueKey: request.uniqueKey,+ forefront,+ };+ }++ // The entry below *is* the write, so a transaction closed during the probe must not receive it -+ // pass through instead, per the closed-transaction rule. Under `deferred` that can land an+ // addition a rollback would have discarded; dedup bounds that cost, silent loss is unbounded.+ if (!transaction.isActive) {+ return await this.addRequest(request, { forefront });+ }++ const snapshot = JSON.parse(JSON.stringify(request)) as Dictionary;++ // Strip-list, not allow-list: every user-facing field flows through, including ones added to+ // `Request` in the future. The exceptions are `id` and `handledAt`, the two backend-owned+ // lifecycle fields.+ delete snapshot.id;+ delete snapshot.handledAt;++ transaction.recordJournalEntry({+ type: 'requestQueue',+ participant: this,+ requests: [{ url: request.url, uniqueKey: request.uniqueKey, label: request.label, snapshot }],+ forefront,+ writeThrough: false,+ });+ buffered.set(request.uniqueKey, snapshot);++ return {+ wasAlreadyPresent: false,+ wasAlreadyHandled: false,+ requestId: getRequestId(request.uniqueKey),+ uniqueKey: request.uniqueKey,+ forefront,+ };+ }++ /** @internal */+ async commitJournalEntries(entries: JournalEntry[]): Promise<void> {+ // Replay through `backend.addBatchOfRequests`, *not* the batched frontend wrapper - the wrapper+ // resolves after the first chunk and sleeps between the rest, neither of which commit may+ // inherit. One call per `forefront` flag; the order of forefront additions is arbitrary anyway.+ for (const forefront of [false, true]) {+ const requests = entries.flatMap((entry) =>+ entry.type === 'requestQueue' && entry.forefront === forefront+ ? // Requests without a snapshot were deduplicated or written through; nothing to replay.+ entry.requests+ .filter((journaled) => journaled.snapshot !== undefined)+ .map((journaled) => new Request(journaled.snapshot as unknown as RequestOptions))+ : [],+ );++ if (requests.length === 0) continue;++ this.statsTracker.add('writeCount');+ const { processedRequests, unprocessedRequests } = await this.backend.addBatchOfRequests(requests, {+ forefront,+ });++ // Only now, with the real backend-assigned ids, may the shared dedup caches be populated.+ for (const processed of processedRequests) {+ const cacheKey = getRequestId(processed.uniqueKey);+ this.cacheRequest(cacheKey, { ...processed, forefront });+ this.requestSeenCache.add(cacheKey, processed.requestId);+ }++ if (unprocessedRequests.length > 0) {+ // Warn and skip, rather than retry or fail. `unprocessedRequests` is what remains after+ // the backend's own transient-error handling - a semantic rejection that retrying here+ // would only re-poke. And failing the commit would let one malformed request hold the+ // whole transaction hostage.+ this.log.warning(+ 'Some requests were rejected by the request queue while committing a storage transaction and will be skipped. ' ++ "This usually means the request data is malformed (e.g. an invalid 'userData' shape).",+ { unprocessedRequests },+ );+ }+ }+ }+/*** Adds requests to the queue in batches of 25. This method will wait till all the requests are added* to the queue before resolving. You should prefer using `queue.addRequestsBatched()` or `crawler.addRequests()`@@ -256,7 +440,7 @@ export class RequestQueue implements IStorage, IRequestManager {requestsLike: RequestsLike,options: RequestQueueOperationOptions = {},): Promise<BatchAddRequestsResult> {- checkStorageAccess();+ const transaction = activeStorageTransaction();ow(requestsLike,@@ -306,6 +490,20 @@ export class RequestQueue implements IStorage, IRequestManager {}}+ if (transaction?.policy.requestQueue === 'deferred') {+ const buffered = this.bufferedRequests(transaction);++ for (const request of requests) {+ results.processedRequests.push(+ await this.addRequestDeferred(transaction, request, forefront, buffered),+ );+ }++ return results;+ }++ this.recordRequestJournalEntry(transaction, requests, forefront, true);+const requestsToAdd = new Map<string, Request>();for (const request of requests) {@@ -370,7 +568,8 @@ export class RequestQueue implements IStorage, IRequestManager {requests: ReadonlyDeep<RequestsLike>,options: AddRequestsBatchedOptions = {},): Promise<AddRequestsBatchedResult> {- checkStorageAccess();+ const transaction = activeStorageTransaction();+ const deferred = transaction?.policy.requestQueue === 'deferred';ow(requests,@@ -428,7 +627,9 @@ export class RequestQueue implements IStorage, IRequestManager {}}- const { batchSize = 1000, waitBetweenBatchesMillis = 1000, maxNewRequests = undefined } = options;+ const { batchSize = 1000, maxNewRequests = undefined } = options;+ // Under `deferred` no chunk performs backend I/O, so pacing them would only stall the handler.+ const waitBetweenBatchesMillis = deferred ? 0 : (options.waitBetweenBatchesMillis ?? 1000);let remainingBudget = maxNewRequests ?? Infinity;const requestsOverLimit: Source[] = [];@@ -512,8 +713,7 @@ export class RequestQueue implements IStorage, IRequestManager {return buildResult(addedRequests, Promise.resolve([]), requestIterator);}- // eslint-disable-next-line no-async-promise-executor- const promise = new Promise<ProcessedRequest[]>(async (resolve) => {+ const processRemainingChunks = async () => {const finalAddedRequests: ProcessedRequest[] = [];for await (const requestChunk of chunks) {@@ -521,7 +721,27 @@ export class RequestQueue implements IStorage, IRequestManager {await sleep(waitBetweenBatchesMillis);}- resolve(finalAddedRequests);+ return finalAddedRequests;+ };++ // maxNewRequests needs all batches to report skipped requests accurately; `deferred` needs them+ // too - a writer that finishes after commit would have nowhere to put its journal entries.+ const awaitsRemainingChunks = options.waitForAllRequestsToBeAdded || maxNewRequests !== undefined || deferred;++ // eslint-disable-next-line no-async-promise-executor+ const promise = new Promise<ProcessedRequest[]>(async (resolve) => {+ if (awaitsRemainingChunks) {+ // Awaited below, i.e. still within the caller's transaction scope, so the additions are+ // journaled like the initial chunk - introspection must not depend on where the chunk+ // boundary happened to fall.+ resolve(await processRemainingChunks());+ } else {+ // Nobody awaits this writer, so it outlives the transaction scope it inherits and must+ // not record into a transaction that may already be closed. It writes directly - its+ // write-through additions were never going to be rolled back anyway - which means the+ // requests it adds are not journaled. See `StorageTransactionView.enqueuedUrls`.+ resolve(await withDirectStorageAccess(processRemainingChunks));+ }});this.inProgressRequestBatchCount += 1;@@ -529,8 +749,7 @@ export class RequestQueue implements IStorage, IRequestManager {this.inProgressRequestBatchCount -= 1;});- // When maxNewRequests is set, we must wait for all batches so we can accurately report skipped requests.- if (options.waitForAllRequestsToBeAdded || maxNewRequests !== undefined) {+ if (awaitsRemainingChunks) {addedRequests.push(...(await promise));}@@ -544,10 +763,16 @@ export class RequestQueue implements IStorage, IRequestManager {* @returns Returns the request object, or `null` if it was not found.*/async getRequest<T extends Dictionary = Dictionary>(uniqueKey: string): Promise<Request<T> | null> {- checkStorageAccess();+ const transaction = activeStorageTransaction();ow(uniqueKey, ow.string);+ // Requests buffered by the active transaction (under the `deferred` write policy) are visible to it.+ const buffered = transaction && this.bufferedRequests(transaction).get(uniqueKey);+ if (buffered) {+ return new Request(buffered as unknown as RequestOptions);+ }+const requestOptions = await this.backend.getRequest(uniqueKey);if (!requestOptions) return null;@@ -572,7 +797,10 @@ export class RequestQueue implements IStorage, IRequestManager {* Returns the request object or `null` if there are no more pending requests.*/async fetchNextRequest<T extends Dictionary = Dictionary>(): Promise<Request<T> | null> {- checkStorageAccess();+ rejectOperationInTransaction(+ 'RequestQueue.fetchNextRequest()',+ 'it is part of the crawler request-processing bookkeeping, which a transaction must not affect.',+ );if (this.queuePausedForMigration) {return null;@@ -592,7 +820,10 @@ export class RequestQueue implements IStorage, IRequestManager {* Handled requests will never again be returned by the `fetchNextRequest` function.*/async markRequestAsHandled(request: Request): Promise<RequestQueueOperationInfo | null> {- checkStorageAccess();+ rejectOperationInTransaction(+ 'RequestQueue.markRequestAsHandled()',+ 'it is part of the crawler request-processing bookkeeping, which a transaction must not affect.',+ );ow(request,@@ -640,7 +871,10 @@ export class RequestQueue implements IStorage, IRequestManager {request: Request,options: RequestQueueOperationOptions = {},): Promise<RequestQueueOperationInfo | null> {- checkStorageAccess();+ rejectOperationInTransaction(+ 'RequestQueue.reclaimRequest()',+ 'it is part of the crawler request-processing bookkeeping, which a transaction must not affect.',+ );ow(request,@@ -687,7 +921,12 @@ export class RequestQueue implements IStorage, IRequestManager {* {@apilink RequestQueue.isFinished}.*/async isEmpty(): Promise<boolean> {- checkStorageAccess();+ const transaction = activeStorageTransaction();++ // Requests buffered by the active transaction count as pending from its point of view.+ if (transaction && this.bufferedRequests(transaction).size > 0) {+ return false;+ }return this.backend.isEmpty();}@@ -701,13 +940,18 @@ export class RequestQueue implements IStorage, IRequestManager {* a false negative, but it shall never return a false positive.*/async isFinished(): Promise<boolean> {- checkStorageAccess();+ const transaction = activeStorageTransaction();// We are not finished if we're still adding new requests in the background.if (this.inProgressRequestBatchCount > 0) {return false;}+ // Requests buffered by the active transaction count as pending from its point of view.+ if (transaction && this.bufferedRequests(transaction).size > 0) {+ return false;+ }+return this.backend.isFinished();}@@ -752,7 +996,7 @@ export class RequestQueue implements IStorage, IRequestManager {* depending on the mode of operation.*/async drop(): Promise<void> {- checkStorageAccess();+ rejectOperationInTransaction('RequestQueue.drop()');await this.backend.drop();serviceLocator.getStorageInstanceManager().removeFromCache(this);@@ -763,7 +1007,7 @@ export class RequestQueue implements IStorage, IRequestManager {* so it can be reused (e.g. across multiple `crawler.run()` calls).*/async purge(): Promise<void> {- checkStorageAccess();+ rejectOperationInTransaction('RequestQueue.purge()');await this.backend.purge();@@ -825,9 +1069,20 @@ export class RequestQueue implements IStorage, IRequestManager {* @throws If the underlying storage no longer exists (e.g. it was deleted externally).*/async getInfo(): Promise<RequestQueueInfo> {- checkStorageAccess();+ const transaction = activeStorageTransaction();++ const metadata = await this.backend.getMetadata();+ const bufferedCount = transaction ? this.bufferedRequests(transaction).size : 0;++ if (bufferedCount > 0) {+ return {+ ...metadata,+ totalRequestCount: metadata.totalRequestCount + bufferedCount,+ pendingRequestCount: metadata.pendingRequestCount + bufferedCount,+ };+ }- return this.backend.getMetadata();+ return metadata;}/**@@ -915,7 +1170,7 @@ export class RequestQueue implements IStorage, IRequestManager {identifier?: string | StorageIdentifier | null,options: StorageOpenOptions = {},): Promise<RequestQueue> {- checkStorageAccess();+ tryCancel();ow(options,diff --git a/packages/core/src/storages/transaction.ts b/packages/core/src/storages/transaction.tsnew file mode 100644index 000000000..b3c8e7800--- /dev/null+++ b/packages/core/src/storages/transaction.ts@@ -0,0 +1,443 @@+import { AsyncLocalStorage } from 'node:async_hooks';++import type { Awaitable, Dictionary } from '@crawlee/types';++import { addTimeoutToPromise, storage as timeoutStorage, tryCancel } from '@apify/timeout';++import { serviceLocator } from '../service_locator.js';+import type { RecordOptions } from './key_value_store.js';++/**+ * Governs whether writes of a given storage type performed inside a {@apilink StorageTransaction} are+ * applied immediately (`writeThrough`) or recorded and replayed on commit (`deferred`).+ */+export type StorageWriteMode = 'deferred' | 'writeThrough';++/**+ * Per-storage-type write policy of a {@apilink StorageTransaction}. Datasets and key-value stores are+ * always `deferred` and not configurable — deferring is the only safe mode for non-idempotent writes,+ * and {@apilink withDirectStorageAccess} covers one-off immediate writes.+ */+export interface StorageWritePolicy {+ /**+ * Write mode for request queue additions. Note that this is a *write policy* for the queue, not the+ * queue instance itself (which is the top-level `requestQueue` crawler option).+ *+ * - `writeThrough` (default): requests are added immediately and are **not** rolled back with the+ * transaction. This is safe (additions are deduplicated by `uniqueKey`, so a retry is idempotent)+ * and keeps new requests visible to the crawler while the handler still runs.+ * - `deferred`: requests are only added when the transaction commits — strict all-or-nothing+ * semantics, at the cost of the crawler not seeing them until the handler finishes.+ */+ requestQueue: StorageWriteMode;+}++const DEFAULT_STORAGE_WRITE_POLICY: StorageWritePolicy = { requestQueue: 'writeThrough' };++const DEFAULT_COMMIT_TIMEOUT_MILLIS = 300_000;++export type StorageTransactionState = 'open' | 'committing' | 'committed' | 'failed' | 'rolledBack';++/**+ * A storage frontend that can record operations in a transaction journal.+ * @internal+ */+export interface TransactionParticipant {+ /**+ * Replay the given buffered journal entries (all recorded by this participant) into the real storage+ * backend. Called during commit, with the transaction already in the `committing` state, so the+ * replayed operations pass through.+ */+ commitJournalEntries(entries: JournalEntry[]): Promise<void>;+}++/**+ * A single dataset write (`pushData`) recorded in a transaction journal.+ */+export interface DatasetJournalEntry {+ type: 'dataset';+ /** @internal **/+ participant: TransactionParticipant;+ storageId: string;+ /** The pushed items, captured by `structuredClone` at write time. */+ items: Dictionary[];+ recordedAt: Date;+}++/**+ * A single key-value store write (`setValue`) recorded in a transaction journal.+ */+export interface KeyValueStoreJournalEntry {+ type: 'keyValueStore';+ /** @internal **/+ participant: TransactionParticipant;+ storageId: string;+ key: string;+ /** The original, pre-serialization value captured by `structuredClone`; `null` denotes a deletion. */+ value: unknown;+ options?: RecordOptions;+}++/**+ * A request recorded in a transaction journal.+ */+export interface JournaledRequest {+ url: string;+ uniqueKey: string;+ label?: string;+ /**+ * A full JSON snapshot of the request for the commit replay. Only present for buffered additions —+ * deduplicated and write-through ones are journaled for introspection only.+ */+ snapshot?: Dictionary;+}++/**+ * A batch of request queue additions recorded in a transaction journal.+ */+export interface RequestQueueJournalEntry {+ type: 'requestQueue';+ /** @internal **/+ participant: TransactionParticipant;+ requests: JournaledRequest[];+ forefront: boolean;+ /** Write-through entries were applied immediately; they are never replayed. */+ writeThrough: boolean;+}++export type JournalEntry = DatasetJournalEntry | KeyValueStoreJournalEntry | RequestQueueJournalEntry;++/**+ * A read-only view of a {@apilink StorageTransaction}: only the journal-backed introspection accessors,+ * without the lifecycle methods. The accessors are synchronous and expose the original pre-serialization+ * values. They cover every write recorded while the transaction was open, under either write policy —+ * with the one exception noted on {@apilink StorageTransactionView.enqueuedUrls|`enqueuedUrls`}. A view+ * is valid until the transaction is disposed.+ */+export interface StorageTransactionView {+ readonly state: StorageTransactionState;++ /** Items pushed to datasets during the transaction, in push order. */+ readonly datasetItems: { item: Dictionary; datasetId: string }[];++ /**+ * URLs enqueued to request queues during the transaction, under either write policy. Recorded as+ * requested, so duplicate, already-present and backend-rejected URLs are included.+ *+ * One gap: unless a caller of `addRequestsBatched()` waits for every chunk+ * (`waitForAllRequestsToBeAdded` or `maxNewRequests`, both of which {@apilink enqueueLinks} sets when+ * a crawl limit applies), the chunks after the first are added by a background writer that outlives+ * the transaction and is not recorded here.+ */+ readonly enqueuedUrls: { url: string; label?: string }[];++ /** Key-value store changes made during the transaction, keyed by store id, last write per key. */+ readonly keyValueStoreChanges: Record<string, Record<string, { changedValue: unknown; options?: RecordOptions }>>;+}++export interface StorageTransactionOptions {+ /** Overrides of the per-storage-type write policy. See {@apilink StorageWritePolicy}. */+ policy?: Partial<StorageWritePolicy>;++ /**+ * How long a commit may take before it fails, in milliseconds. There is no automatic retry — the+ * replay of dataset items is not idempotent.+ * @default 300000+ */+ commitTimeoutMillis?: number;+}++const transactionStorage = new AsyncLocalStorage<StorageTransaction>();++const COMMIT_ORDER: JournalEntry['type'][] = ['keyValueStore', 'requestQueue', 'dataset'];++/**+ * A storage transaction scoped to a request's lifecycle. Writes made through the storage frontends+ * ({@apilink Dataset}, {@apilink KeyValueStore}, {@apilink RequestQueue}) while the transaction is active+ * are recorded rather than applied; on {@apilink StorageTransaction.commit|`commit()`} they are replayed+ * into real storage, on {@apilink StorageTransaction.rollback|`rollback()`} they are dropped. Reads consult+ * the recorded writes first, so a handler sees its own writes.+ *+ * Create one with {@apilink createStorageTransaction} (explicit commit/rollback) or+ * {@apilink withStorageTransaction} (scoped sugar). Crawlers open one automatically around every request+ * handler unless `transactionalStorage: false` is set.+ */+export class StorageTransaction implements StorageTransactionView {+ /** The ordered, append-only journal — the source of truth for commit, introspection and reads. */+ readonly journal: JournalEntry[] = [];++ /** Per-storage-type write policy. */+ readonly policy: StorageWritePolicy;++ private readonly commitTimeoutMillis: number;++ private _state: StorageTransactionState = 'open';++ private disposed = false;++ /** @internal */+ constructor(options: StorageTransactionOptions = {}) {+ this.policy = { ...DEFAULT_STORAGE_WRITE_POLICY, ...options.policy };+ this.commitTimeoutMillis = options.commitTimeoutMillis ?? DEFAULT_COMMIT_TIMEOUT_MILLIS;+ }++ get state(): StorageTransactionState {+ return this._state;+ }++ /**+ * `true` only while `state === 'open'`. This is the single predicate every storage operation+ * consults — operations performed after the transaction is closed pass through to the real backend.+ */+ get isActive(): boolean {+ return this._state === 'open';+ }++ /** Runs `callback` with this transaction installed in the async context. */+ async run<T>(callback: () => Awaitable<T>): Promise<T> {+ return transactionStorage.run(this, async () => callback());+ }++ /**+ * Records a write operation in the journal.+ * @internal+ */+ recordJournalEntry(entry: JournalEntry): void {+ if (!this.isActive) {+ throw new Error(`Cannot record a journal entry on a transaction in the '${this._state}' state`);+ }++ this.journal.push(entry);+ }++ /**+ * Replays the journaled writes into real storage. A no-op unless the transaction is `open`.+ *+ * The transaction transitions to `committing` *before* anything is flushed, so a commit that throws+ * partway lands in `failed` (never back in `open`) and subsequent storage operations pass through+ * rather than recording into a dead transaction. Delivery is at-least-once — a commit that fails+ * partway may have applied some of the writes already.+ */+ async commit(): Promise<void> {+ if (this._state !== 'open') {+ return;+ }++ this._state = 'committing';++ try {+ // The replay re-drives the frontend write path, which checks for cancellation (`tryCancel`)+ // on every operation - and `@apify/timeout` shares one `AbortController` across nested+ // frames, so a request-handler timeout that already fired would abort the commit of a+ // handler that succeeded. Hence a fresh timeout context, which also provides the time bound.+ await timeoutStorage.exit(async () =>+ addTimeoutToPromise(+ async () => this.flush(),+ this.commitTimeoutMillis,+ `Committing the storage transaction timed out after ${this.commitTimeoutMillis / 1000} seconds.`,+ ),+ );+ this._state = 'committed';+ } catch (error) {+ this._state = 'failed';+ throw error;+ }+ }++ private async flush(): Promise<void> {+ // Each participating frontend replays all of its buffered entries in one call. Frontends are+ // ordered by storage type: key-value stores and request queues first (idempotent under retry),+ // datasets last (not idempotent), minimizing the blast radius of a partial commit failure.+ const groups = new Map<TransactionParticipant, JournalEntry[]>();++ for (const entry of this.journal) {+ if (entry.type === 'requestQueue' && entry.writeThrough) continue;+ const group = groups.get(entry.participant);+ if (group) group.push(entry);+ else groups.set(entry.participant, [entry]);+ }++ // A participant only records entries of its own storage type, so the first entry determines+ // the group's place in the commit order.+ const orderedGroups = [...groups.values()].sort(+ (a, b) => COMMIT_ORDER.indexOf(a[0].type) - COMMIT_ORDER.indexOf(b[0].type),+ );++ for (const entries of orderedGroups) {+ await entries[0].participant.commitJournalEntries(entries);+ }+ }++ /**+ * Discards the journaled writes. A no-op unless the transaction is `open` — in particular, calling it+ * after a successful `commit()` (which the crawler's error handling can legitimately do) does nothing+ * and never throws.+ */+ rollback(): void {+ if (this._state !== 'open') {+ return;+ }++ this._state = 'rolledBack';+ }++ /**+ * Releases the journal and the write-time snapshots it holds. Must be called for *every* terminal+ * state, `failed` included. Idempotent, never throws, and does not change `state`. Any+ * {@apilink StorageTransactionView} of this transaction is only valid until this is called.+ */+ dispose(): void {+ if (this.disposed) {+ return;+ }++ if (this._state === 'open') {+ // Disposing an open transaction is an internal invariant violation - roll back first.+ try {+ serviceLocator+ .getLogger()+ .warning('Internal error: a storage transaction was disposed while still open; rolling it back.');+ } catch {+ // Never throw from dispose.+ }+ this.rollback();+ }++ this.disposed = true;+ this.journal.length = 0;+ }++ get datasetItems(): { item: Dictionary; datasetId: string }[] {+ return this.journal.flatMap((entry) =>+ entry.type === 'dataset' ? entry.items.map((item) => ({ item, datasetId: entry.storageId })) : [],+ );+ }++ get enqueuedUrls(): { url: string; label?: string }[] {+ return this.journal.flatMap((entry) =>+ entry.type === 'requestQueue' ? entry.requests.map(({ url, label }) => ({ url, label })) : [],+ );+ }++ get keyValueStoreChanges(): Record<string, Record<string, { changedValue: unknown; options?: RecordOptions }>> {+ const result: Record<string, Record<string, { changedValue: unknown; options?: RecordOptions }>> = {};++ for (const entry of this.journal) {+ if (entry.type !== 'keyValueStore') continue;+ result[entry.storageId] ??= {};+ result[entry.storageId][entry.key] = { changedValue: entry.value, options: entry.options };+ }++ return result;+ }+}++/**+ * Opens a {@apilink StorageTransaction} without running anything yet. The caller owns the outcome:+ * `run()`, then `commit()` or `rollback()`, and always `dispose()` when done. For the common+ * open-run-commit flow, prefer {@apilink withStorageTransaction}.+ */+export function createStorageTransaction(options: StorageTransactionOptions = {}): StorageTransaction {+ return new StorageTransaction(options);+}++/**+ * Runs `callback` inside a new {@apilink StorageTransaction}: storage writes made in the callback are+ * committed when it returns and rolled back when it throws. If a transaction is already active in the+ * current async context, it is reused and its outcome is left to its owner (and `options` are ignored)+ * — there are no nested transaction semantics.+ */+export async function withStorageTransaction<T>(+ callback: (transaction: StorageTransaction) => Awaitable<T>,+ options: StorageTransactionOptions = {},+): Promise<T> {+ const existing = transactionStorage.getStore();+ if (existing?.isActive) {+ return callback(existing);+ }++ const transaction = createStorageTransaction(options);++ try {+ const result = await transaction.run(async () => callback(transaction));+ await transaction.commit();+ return result;+ } catch (error) {+ transaction.rollback();+ throw error;+ } finally {+ transaction.dispose();+ }+}++/**+ * Runs `callback` outside of any storage transaction — the per-call-site escape hatch. Storage operations+ * made inside it hit the real backend directly, are not rolled back, and operations that a transaction+ * rejects (`drop`, stream-valued `setValue`, request queue internals, ...) are permitted.+ */+export async function withDirectStorageAccess<T>(callback: () => Awaitable<T>): Promise<T> {+ return transactionStorage.exit(async () => callback());+}++/**+ * The per-operation hook consulted by every storage frontend method: performs the cancellation check+ * that aborts storage operations when the request handler times out, and returns the active storage+ * transaction. Returns `undefined` when there is no transaction in the async context *or* when it is no+ * longer open — operations on a closed transaction deliberately pass through to the real backend.+ * @internal+ */+export function activeStorageTransaction(): StorageTransaction | undefined {+ tryCancel();+ const transaction = transactionStorage.getStore();+ return transaction?.isActive ? transaction : undefined;+}++/**+ * Returns the transaction installed in the current async context, regardless of its state. Used by the+ * crawler to drive the outcome of the transaction it opened.+ * @internal+ */+export function currentStorageTransaction(): StorageTransaction | undefined {+ return transactionStorage.getStore();+}++/**+ * Captures a value at write time, so that later mutations of the caller's object affect neither the+ * read-your-own-writes reads nor the commit replay. `structuredClone` for fidelity (`Date`, `Map`, `Set`,+ * typed arrays, `undefined`); values it cannot handle fall back to the JSON round-trip the storage+ * backends perform anyway.+ * @internal+ */+export function snapshotValue<T>(value: T): T {+ try {+ return structuredClone(value);+ } catch {+ return JSON.parse(JSON.stringify(value));+ }+}++/**+ * The guard for operations that cannot be performed inside a storage transaction: throws when one is+ * active, and performs the per-operation cancellation check either way.+ * @internal+ */+export function rejectOperationInTransaction(operation: string, reason = 'it cannot be rolled back.'): void {+ if (activeStorageTransaction() === undefined) {+ return;+ }++ throw operationRejectedInTransaction(operation, reason);+}++/**+ * Builds the "operation not allowed in a transaction" error, for a call site that has already+ * established a transaction is active and so wants to `throw` unconditionally.+ * @internal+ */+export function operationRejectedInTransaction(operation: string, reason = 'it cannot be rolled back.'): Error {+ return new Error(+ `${operation} cannot be used inside a storage transaction: ${reason} ` ++ 'If you really need it, wrap the call in withDirectStorageAccess(() => ...) - operations ' ++ 'performed there are applied immediately and are not rolled back.',+ );+}diff --git a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.tsindex c4aa91de2..87a57a820 100644--- a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts+++ b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts@@ -19,27 +19,28 @@ import type {CrawlingContext,EnqueueLinksOptions,GetUserDataFromRequest,- RequestQueue,RestrictedCrawlingContext,RouterRoutes,StatisticPersistedState,StatisticsOptions,StatisticState,+ StorageTransaction,+ StorageTransactionView,+ StorageWritePolicy,} from '@crawlee/core';import {+ createStorageTransaction,OwnedOrInjected,RequestHandlerError,- RequestHandlerResult,resolveBaseUrlForEnqueueLinksFiltering,Router,- serviceLocator,Statistics,- withCheckedStorageAccess,} from '@crawlee/core';import type { BatchAddRequestsResult, Dictionary, Awaitable } from '@crawlee/types';import { type CheerioRoot, extractUrlsFromCheerio } from '@crawlee/utils';import { type Cheerio } from 'cheerio';import type { AnyNode } from 'domhandler';+import ow from 'ow';import type { Page } from 'playwright';import type { SetRequired } from 'type-fest';@@ -224,11 +225,12 @@ export interface AdaptivePlaywrightCrawlerOptions<renderingTypeDetectionRatio?: number;/**- * An optional callback that is called on dataset items found by the request handler in plain HTTP mode.+ * An optional callback that is called on the storage writes recorded by the request handler in plain+ * HTTP mode (exposed as a read-only {@apilink StorageTransactionView}).* If it returns false, the request is retried in a browser.- * If no callback is specified, every dataset item is considered valid.+ * If no callback is specified, every result is considered valid.*/- resultChecker?: (result: RequestHandlerResult) => boolean;+ resultChecker?: (result: StorageTransactionView) => boolean;/*** An optional callback that decides whether an error thrown during the plain HTTP request handler@@ -254,8 +256,8 @@ export interface AdaptivePlaywrightCrawlerOptions<* For a stricter, ready-made comparator that also takes enqueued requests and key-value store changes into account, see {@apilink fullResultComparator}.*/resultComparator?: (- resultA: RequestHandlerResult,- resultB: RequestHandlerResult,+ resultA: StorageTransactionView,+ resultB: StorageTransactionView,) => boolean | 'equal' | 'different' | 'inconclusive';/**@@ -264,12 +266,6 @@ export interface AdaptivePlaywrightCrawlerOptions<* Omit the option and the crawler builds its own from `renderingTypeDetectionRatio` - and initializes it.*/renderingTypePredictor?: IRenderingTypePredictor;-- /**- * Prevent direct access to storage in request handlers (only allow using context helpers).- * Defaults to `true`- */- preventDirectStorageAccess?: boolean;}const proxyLogMethods = [@@ -326,7 +322,6 @@ export class AdaptivePlaywrightCrawler<private resultChecker: NonNullable<AdaptivePlaywrightCrawlerOptions['resultChecker']>;private shouldPropagateError: NonNullable<AdaptivePlaywrightCrawlerOptions['shouldPropagateError']>;private resultComparator: NonNullable<AdaptivePlaywrightCrawlerOptions['resultComparator']>;- private preventDirectStorageAccess: boolean;private staticContextPipeline: ContextPipeline<CrawlingContext, ExtendedContext>;private browserContextPipeline: ContextPipeline<CrawlingContext, ExtendedContext>;private individualRequestHandlerTimeoutMillis: number;@@ -335,9 +330,15 @@ export class AdaptivePlaywrightCrawler<override get stats(): AdaptivePlaywrightCrawlerStatistics {return super.stats as AdaptivePlaywrightCrawlerStatistics;}- private resultObjects = new WeakMap<CrawlingContext, RequestHandlerResult>();+private inFlightRenderingTypeDetections = 0;+ /**+ * The write policy of the per-attempt transactions. Defaults the request queue to `deferred`:+ * a discarded attempt's enqueues must never reach the queue.+ */+ private readonly attemptWritePolicy: Partial<StorageWritePolicy>;+private teardownHooks: (() => Promise<unknown>)[] = [];constructor(options: AdaptivePlaywrightCrawlerOptions<ContextExtension, ExtendedContext, Routes> = {}) {@@ -349,7 +350,6 @@ export class AdaptivePlaywrightCrawler<shouldPropagateError,resultComparator,statistics,- preventDirectStorageAccess = true,requestHandlerTimeoutSecs = 60,errorHandler,failedRequestHandler,@@ -357,9 +357,24 @@ export class AdaptivePlaywrightCrawler<postNavigationHooks = [],extendContext,contextPipelineBuilder,+ transactionalStorage,...rest} = options;+ // The user's value is replaced by `false` in the `super` call below — validate it separately.+ ow(transactionalStorage, 'transactionalStorage', BasicCrawler.optionsShape.transactionalStorage);++ // Per-attempt buffering is load-bearing here: the handler runs up to twice per request and the+ // losing attempt's writes must be discardable.+ if (transactionalStorage === false) {+ throw new Error(+ 'AdaptivePlaywrightCrawler requires transactional storage - it runs the request handler ' ++ 'multiple times per request and must be able to discard the storage writes of losing ' ++ 'attempts. `transactionalStorage: false` is therefore not supported; a write policy ' ++ 'object is accepted and forwarded to the per-attempt transactions.',+ );+ }+if (statistics !== undefined && !(statistics instanceof AdaptivePlaywrightCrawlerStatistics)) {throw new Error('AdaptivePlaywrightCrawler tracks extra fields on its own Statistics subclass and cannot use a ' +@@ -380,6 +395,10 @@ export class AdaptivePlaywrightCrawler<logMessage: `${AdaptivePlaywrightCrawler.name} request statistics:`,}),contextPipelineBuilder: contextPipelineBuilder ?? (() => this.buildContextPipeline()),+ // The base crawler must not wrap requests in a transaction of its own - this crawler opens+ // one per request handler attempt in `crawlOne` instead, forwarding the write policy of the+ // user-facing option (validated above) to those.+ transactionalStorage: false,});this.individualRequestHandlerTimeoutMillis = requestHandlerTimeoutSecs * 1000;@@ -389,6 +408,11 @@ export class AdaptivePlaywrightCrawler<renderingTypePredictor,() => new RenderingTypePredictor({ detectionRatio: renderingTypeDetectionRatio }),);+ this.attemptWritePolicy = {+ requestQueue: 'deferred',+ ...(typeof transactionalStorage === 'object' ? transactionalStorage : {}),+ };+this.resultChecker = resultChecker ?? (() => true);this.shouldPropagateError = shouldPropagateError ?? (() => false);@@ -440,8 +464,6 @@ export class AdaptivePlaywrightCrawler<this.browserContextPipeline = browserCrawler.contextPipeline.compose({action: this.adaptPlaywrightContext.bind(this),}) as unknown as ContextPipeline<CrawlingContext, ExtendedContext>;-- this.preventDirectStorageAccess = preventDirectStorageAccess;}protected override async _init(): Promise<void> {@@ -483,12 +505,6 @@ export class AdaptivePlaywrightCrawler<}private async adaptCheerioContext(cheerioContext: CheerioCrawlingContext) {- // Capture the original response to avoid infinite recursion when the getter is copied to the context- const result = this.resultObjects.get(cheerioContext);- if (result === undefined) {- throw new Error('Logical error - `this.resultObjects` does not contain the result object');- }-return {get page(): Page {throw new Error('Page object was used in HTTP-only request handler');@@ -507,24 +523,16 @@ export class AdaptivePlaywrightCrawler<options.selector,options.baseUrl ?? cheerioContext.request.loadedUrl,);- return (await this.enqueueLinks(- { ...options, urls },- cheerioContext.request,- result,- )) as unknown as void;+ return (await this.enqueueLinks({ ...options, urls }, cheerioContext.request)) as unknown as void;},response: cheerioContext.response,};}private async adaptPlaywrightContext(playwrightContext: PlaywrightCrawlingContext) {+ // Capture the original response to avoid infinite recursion when the getter is copied to the contextconst originalResponse = playwrightContext.response;- const result = this.resultObjects.get(playwrightContext);- if (result === undefined) {- throw new Error('Logical error - `this.resultObjects` does not contain the result object');- }-return {response: new Response(Uint8Array.from(await originalResponse.body()), {headers: originalResponse.headers(),@@ -564,33 +572,35 @@ export class AdaptivePlaywrightCrawler<urls = options.urls;}- return (await this.enqueueLinks(- { ...options, urls },- playwrightContext.request,- result,- )) as unknown as void;+ return (await this.enqueueLinks({ ...options, urls }, playwrightContext.request)) as unknown as void;},};}+ /**+ * Runs one request handler attempt inside its own {@apilink StorageTransaction}, wrapping the inner+ * (static or browser) context pipeline. The transaction is pushed to `transactions` *at creation+ * time, before the `try`* - the `ok: false` branch of the returned {@apilink Result} carries no+ * result, and failed attempts are routine here. The caller owns the outcome and disposal.+ */private async crawlOne(renderingType: RenderingType,context: CrawlingContext,useStateFunction: (defaultValue?: Dictionary) => Promise<Dictionary>,- ): Promise<Result<RequestHandlerResult>> {- const result = new RequestHandlerResult(- serviceLocator.getConfiguration(),- AdaptivePlaywrightCrawler.CRAWLEE_STATE_KEY,- );+ transactions: StorageTransaction[],+ ): Promise<Result<StorageTransaction>> {+ const transaction = createStorageTransaction({+ policy: this.attemptWritePolicy,+ commitTimeoutMillis: this.internalTimeoutMillis,+ });+ transactions.push(transaction);+const logs: LogProxyCall[] = [];const deferredCleanup: (() => Promise<unknown>)[] = [];- const resultBoundContextHelpers = {- addRequests: result.addRequests,- pushData: result.pushData,- useState: this.allowStorageAccess(useStateFunction),- getKeyValueStore: this.allowStorageAccess(result.getKeyValueStore),+ const attemptBoundContextHelpers = {+ useState: useStateFunction,log: this.createLogProxy(context.log, logs),registerDeferredCleanup: (cleanup: () => Promise<unknown>) => deferredCleanup.push(cleanup),};@@ -600,14 +610,12 @@ export class AdaptivePlaywrightCrawler<Object.getOwnPropertyDescriptors(context),) as typeof context;- // Mark result-bound helpers as non-configurable so they survive the sub-crawler context pipeline- // (which would otherwise override them with the sub-crawler's own versions, losing the result binding).- for (const [key, descriptor] of Object.entries(Object.getOwnPropertyDescriptors(resultBoundContextHelpers))) {+ // Mark attempt-bound helpers as non-configurable so they survive the sub-crawler context pipeline+ // (which would otherwise override them with the sub-crawler's own versions, losing the binding).+ for (const [key, descriptor] of Object.entries(Object.getOwnPropertyDescriptors(attemptBoundContextHelpers))) {Object.defineProperty(subCrawlerContext, key, { ...descriptor, configurable: false });}- this.resultObjects.set(subCrawlerContext, result);-try {const callAdaptiveRequestHandler = async () => {if (renderingType === 'static') {@@ -626,19 +634,12 @@ export class AdaptivePlaywrightCrawler<routeTimeoutSecs === undefined ? this.individualRequestHandlerTimeoutMillis : routeTimeoutSecs * 1000;await addTimeoutToPromise(- async () =>- withCheckedStorageAccess(() => {- if (this.preventDirectStorageAccess) {- throw new Error(- 'Directly accessing storage in a request handler is not allowed in AdaptivePlaywrightCrawler',- );- }- }, callAdaptiveRequestHandler),+ async () => transaction.run(callAdaptiveRequestHandler),timeoutMillis,'Request handler timed out',);- return { result, ok: true, logs };+ return { result: transaction, ok: true, logs };} catch (error) {return { error, ok: false, logs };} finally {@@ -660,17 +661,27 @@ export class AdaptivePlaywrightCrawler<this.inFlightRenderingTypeDetections += 1;}+ // Every transaction created for this request - up to two, since the static-then-browser+ // fall-through and the browser-then-detection pair are mutually exclusive. Disposed in the+ // `finally` below, not earlier: the comparators read the journals after `crawlOne` returns.+ const transactions: StorageTransaction[] = [];+try {if (renderingTypePrediction.renderingType === 'static' && !shouldDetectRenderingType) {crawlingContext.log.debug(`Running HTTP-only request handler for ${crawlingContext.request.url}`);this.stats.trackHttpOnlyRequestHandlerRun();- const plainHTTPRun = await this.crawlOne('static', crawlingContext, crawlingContext.useState);+ const plainHTTPRun = await this.crawlOne(+ 'static',+ crawlingContext,+ crawlingContext.useState,+ transactions,+ );if (plainHTTPRun.ok && this.resultChecker(plainHTTPRun.result)) {crawlingContext.log.debug(`HTTP-only request handler succeeded for ${crawlingContext.request.url}`);plainHTTPRun.logs?.forEach(([log, method, ...args]) => log[method](...(args as [any, any])));- await this.commitResult(crawlingContext, plainHTTPRun.result);+ await plainHTTPRun.result.commit();return;}@@ -727,6 +738,7 @@ export class AdaptivePlaywrightCrawler<'clientOnly',crawlingContext,stateTracker.getLiveState.bind(stateTracker),+ transactions,);if (!browserRun.ok) {@@ -734,14 +746,17 @@ export class AdaptivePlaywrightCrawler<}browserRun.logs?.forEach(([log, method, ...args]) => log[method](...(args as [any, any])));- await this.commitResult(crawlingContext, browserRun.result);+ await browserRun.result.commit();if (shouldDetectRenderingType) {crawlingContext.log.debug(`Detecting rendering type for ${crawlingContext.request.url}`);+ // The detection attempt's transaction is never committed - its writes exist only for the+ // result comparison.const plainHTTPRun = await this.crawlOne('static',crawlingContext,stateTracker.getStateCopy.bind(stateTracker),+ transactions,);const detectionResult: RenderingType | undefined = (() => {@@ -773,50 +788,18 @@ export class AdaptivePlaywrightCrawler<if (shouldDetectRenderingType) {this.inFlightRenderingTypeDetections -= 1;}- }- }-- private async commitResult(- crawlingContext: CrawlingContext,- { calls, keyValueStoreChanges }: RequestHandlerResult,- ): Promise<void> {- await Promise.all([- ...calls.pushData.map(async (params) => crawlingContext.pushData(...params)),- ...calls.addRequests.map(async (params) => crawlingContext.addRequests(...params)),- ...Object.entries(keyValueStoreChanges).map(async ([storeIdOrName, changes]) => {- const store = await crawlingContext.getKeyValueStore({ id: storeIdOrName });- await Promise.all(- Object.entries(changes).map(async ([key, { changedValue, options }]) =>- store.setValue(key, changedValue, options),- ),- );- }),- ]);- }-- private allowStorageAccess<R, TArgs extends any[]>(- func: (...args: TArgs) => Promise<R>,- ): (...args: TArgs) => Promise<R> {- return async (...args: TArgs) =>- withCheckedStorageAccess(- () => {},- async () => func(...args),- );- }- /**- * Reading the pending request count queries the underlying request manager, which counts as storage access.- * Since this is internal crawler bookkeeping used to compute the `enqueueLinks` limit (not user-initiated storage- * access), it must be allowed even while a request handler runs inside the storage-access guard.- */- protected override async getPendingRequestCountApproximation(): Promise<number> {- return this.allowStorageAccess(() => super.getPendingRequestCountApproximation())();+ // A still-open transaction here belongs to a discarded attempt - roll it back, then release.+ for (const transaction of transactions) {+ transaction.rollback();+ transaction.dispose();+ }+ }}private async enqueueLinks(options: SetRequired<EnqueueLinksOptions, 'urls'>,request: RestrictedCrawlingContext['request'],- result: RequestHandlerResult,): Promise<BatchAddRequestsResult> {const baseUrl = resolveBaseUrlForEnqueueLinksFiltering({enqueueStrategy: options?.strategy,@@ -825,24 +808,9 @@ export class AdaptivePlaywrightCrawler<userProvidedBaseUrl: options?.baseUrl,});- const addRequestsBatched: RequestQueue['addRequestsBatched'] = async (requests: Request<Dictionary>[]) => {- await result.addRequests(requests);-- return {- addedRequests: requests.map(({ uniqueKey, id }) => ({- uniqueKey,- requestId: id ?? '',- wasAlreadyPresent: false,- wasAlreadyHandled: false,- })),- waitForAllRequestsToBeAdded: Promise.resolve([]),- requestsOverLimit: [],- };- };- // We need to use a mock request queue implementation, in order to add the requests into our result object- const mockRequestQueue = { addRequestsBatched } as RequestQueue;-- return await this.enqueueLinksWithCrawlDepth({ ...options, baseUrl }, request, mockRequestQueue);+ // The per-attempt transaction buffers these (the queue policy defaults to `deferred` here),+ // so a discarded attempt's enqueues never reach the queue.+ return await this.enqueueLinksWithCrawlDepth({ ...options, baseUrl }, request, await this.getRequestManager());}private createLogProxy(log: CrawleeLogger, logs: LogProxyCall[]) {@@ -905,11 +873,10 @@ export function createAdaptivePlaywrightRouter(routesOrSchemas?: any): any {* });* ```*/-export function fullResultComparator(resultA: RequestHandlerResult, resultB: RequestHandlerResult): boolean {+export function fullResultComparator(resultA: StorageTransactionView, resultB: StorageTransactionView): boolean {return (isDeepStrictEqual(resultA.datasetItems, resultB.datasetItems) &&isDeepStrictEqual(resultA.enqueuedUrls, resultB.enqueuedUrls) &&- isDeepStrictEqual(resultA.enqueuedUrlLists, resultB.enqueuedUrlLists) &&isDeepStrictEqual(resultA.keyValueStoreChanges, resultB.keyValueStoreChanges));}diff --git a/test/core/crawlers/adaptive_playwright_crawler.test.ts b/test/core/crawlers/adaptive_playwright_crawler.test.tsindex 119c9731a..678cd7ce4 100644--- a/test/core/crawlers/adaptive_playwright_crawler.test.ts+++ b/test/core/crawlers/adaptive_playwright_crawler.test.ts@@ -315,9 +315,21 @@ describe('AdaptivePlaywrightCrawler', () => {},);- const crawler = await makeOneshotCrawler({ requestHandler, renderingTypePredictor, ...options }, [- url.toString(),- ]);+ const crawler = await makeOneshotCrawler(+ {+ requestHandler,+ renderingTypePredictor,+ // The enqueue budget derived from `maxRequestsPerCrawl` applies to the transaction+ // journal too (only requests that would really be enqueued are recorded), so the seed+ // request needs room for its five links. The crawl is stopped right after the seed's+ // rendering type detection (see below), so the enqueued links are never crawled.+ maxRequestsPerCrawl: 10,+ ...options,+ },+ [url.toString()],+ );++ renderingTypePredictor.storeResult.mockImplementation(() => void crawler.stop());return { crawler, renderingTypePredictor };};@@ -722,7 +734,17 @@ describe('AdaptivePlaywrightCrawler', () => {await expect(store.getValue('3')).resolves.toEqual({ content: 42 });});- test('should not allow direct key-value store manipulation', async () => {+ test('should reject transactionalStorage: false at construction', () => {+ expect(+ () =>+ new AdaptivePlaywrightCrawler({+ transactionalStorage: false,+ requestHandler: async () => {},+ }),+ ).toThrow(/requires transactional storage/);+ });++ test('should capture direct key-value store manipulation and commit it with the winning attempt', async () => {const renderingTypePredictor = makeRiggedRenderingTypePredictor({detectionProbabilityRecommendation: 0,renderingType: 'static',@@ -747,13 +769,10 @@ describe('AdaptivePlaywrightCrawler', () => {);await crawler.run();- expect(failedRequestHandler.mock.calls).toHaveLength(1);- expect((failedRequestHandler.mock.calls[0][1] as Error).message).toEqual(- 'Directly accessing storage in a request handler is not allowed in AdaptivePlaywrightCrawler',- );+ expect(failedRequestHandler).not.toHaveBeenCalled();const store = await KeyValueStore.open();- expect(await store.getValue('1')).toBeNull();+ expect(await store.getValue('1')).toEqual({ content: 42 });});test('should persist RenderingTypePredictor state on PERSIST_STATE events', async () => {diff --git a/test/core/crawlers/basic_crawler.test.ts b/test/core/crawlers/basic_crawler.test.tsindex 2dbe040e6..3f98cc951 100644--- a/test/core/crawlers/basic_crawler.test.ts+++ b/test/core/crawlers/basic_crawler.test.ts@@ -9,6 +9,7 @@ import {BasicCrawler,Configuration,CriticalError,+ Dataset,defaultRoute,ErrorTracker,EventType,@@ -3036,6 +3037,204 @@ describe('BasicCrawler', () => {});});+ describe('transactional storage', () => {+ test('a failing request handler leaves no dataset/KVS writes, but write-through enqueues survive', async () => {+ const crawler = new BasicCrawler({+ maxRequestRetries: 0,+ requestHandler: async ({ request, pushData, addRequests }) => {+ if (request.label === 'FAIL') {+ await pushData({ from: 'failing-handler' });+ await (await KeyValueStore.open()).setValue('failing-key', { a: 1 });+ await addRequests([{ url: `http://${HOSTNAME}:${port}/child`, label: 'CHILD' }]);+ throw new Error('handler failed');+ }++ await pushData({ from: 'child-handler' });+ },+ });++ await crawler.run([{ url: `http://${HOSTNAME}:${port}/`, label: 'FAIL' }]);++ // The write-through enqueue survived the failure and the child was crawled...+ const dataset = await Dataset.open();+ await expect(dataset.getData()).resolves.toMatchObject({ items: [{ from: 'child-handler' }] });++ // ...while the failing handler's other writes were rolled back.+ await expect(KeyValueStore.getValue('failing-key')).resolves.toBeNull();+ });++ test('a retried handler does not double-write, and the deferred queue policy reaches the transaction', async () => {+ const crawler = new BasicCrawler({+ maxRequestRetries: 1,+ transactionalStorage: { requestQueue: 'deferred' },+ requestHandler: async ({ request, pushData, addRequests }) => {+ if (request.label === 'CHILD') {+ return;+ }++ await pushData({ attempt: request.retryCount });+ // A different child per attempt, so `uniqueKey` dedup cannot mask a write-through+ // enqueue of the failed attempt.+ await addRequests([+ { url: `http://${HOSTNAME}:${port}/child-${request.retryCount}`, label: 'CHILD' },+ ]);++ if (request.retryCount === 0) {+ throw new Error('first attempt fails');+ }+ },+ });++ await crawler.run([`http://${HOSTNAME}:${port}/`]);++ // Exactly one copy of the data, from the successful attempt.+ const dataset = await Dataset.open();+ await expect(dataset.getData()).resolves.toMatchObject({ total: 1, items: [{ attempt: 1 }] });++ // Only the successful attempt's enqueue reached the queue: the seed plus `child-1`.+ // A write-through enqueue of the failed attempt would have left `child-0` behind as well.+ await expect((await crawler.getRequestQueue()).getTotalCount()).resolves.toBe(2);+ });++ test('errorHandler and failedRequestHandler writes reach real storage', async () => {+ const crawler = new BasicCrawler({+ maxRequestRetries: 1,+ requestHandler: async ({ pushData }) => {+ await pushData({ from: 'handler' });+ throw new Error('always fails');+ },+ errorHandler: async () => {+ await (await KeyValueStore.open()).setValue('error-handler', { ran: true });+ },+ failedRequestHandler: async () => {+ await (await KeyValueStore.open()).setValue('failed-request-handler', { ran: true });+ },+ });++ await crawler.run([`http://${HOSTNAME}:${port}/`]);++ await expect(KeyValueStore.getValue('error-handler')).resolves.toEqual({ ran: true });+ await expect(KeyValueStore.getValue('failed-request-handler')).resolves.toEqual({ ran: true });++ const dataset = await Dataset.open();+ await expect(dataset.getData()).resolves.toMatchObject({ total: 0 });+ });++ test('transactionalStorage: false disables the mechanism entirely', async () => {+ const crawler = new BasicCrawler({+ maxRequestRetries: 0,+ transactionalStorage: false,+ requestHandler: async ({ pushData }) => {+ await pushData({ from: 'failing-handler' });+ throw new Error('handler failed');+ },+ });++ await crawler.run([`http://${HOSTNAME}:${port}/`]);++ // Without transactions, the write of the failing handler lands immediately and stays.+ const dataset = await Dataset.open();+ await expect(dataset.getData()).resolves.toMatchObject({ items: [{ from: 'failing-handler' }] });+ });++ test('an unclosed transaction on a normal pipeline return is discarded and logged', async () => {+ const crawler = new BasicCrawler({ requestHandler: async () => {} });+ const errorSpy = vitest.spyOn((crawler as any).log, 'error').mockImplementation(() => {});++ // Simulate the wiring bug the guard exists for: the pipeline callback returns normally while+ // its transaction is still open (handleRequest failed to commit or roll it back).+ let leaked: any;+ await (crawler as any).runInStorageTransaction(async () => {+ leaked = (await import('@crawlee/core')).currentStorageTransaction();+ });++ expect(leaked.state).toBe('rolledBack'); // discarded, not left open+ expect(errorSpy).toHaveBeenCalledWith(expect.stringMatching(/still open after the request pipeline/));++ errorSpy.mockRestore();+ });++ test('the crawler never fetches the next request inside a storage transaction', async () => {+ // The invariant that lets the request manager stay ignorant of transactions: fetching is+ // crawler bookkeeping that runs *before* the request's transaction opens. If a refactor ever+ // moved the fetch inside the transaction scope, a buffered add/transfer could be rolled back+ // after the source request was consumed - losing the request. Observed through the public+ // `requestManager` option, so the assertion sits on a documented seam, not crawler internals.+ const { currentStorageTransaction } = await import('@crawlee/core');++ const realManager = await RequestQueue.open();+ await realManager.addRequests([+ `http://${HOSTNAME}:${port}/one`,+ `http://${HOSTNAME}:${port}/two`,+ `http://${HOSTNAME}:${port}/three`,+ ]);++ let fetches = 0;+ let fetchedInsideTransaction = false;++ // A pass-through manager: delegates everything to a real queue, only observing whether a+ // transaction is active at fetch time.+ const observingManager = new Proxy(realManager, {+ get(target, prop, receiver) {+ if (prop === 'fetchNextRequest') {+ return async (...args: unknown[]) => {+ fetches += 1;+ if (currentStorageTransaction()?.isActive) fetchedInsideTransaction = true;+ return (target.fetchNextRequest as (...a: unknown[]) => unknown)(...args);+ };+ }+ const value = Reflect.get(target, prop, receiver);+ return typeof value === 'function' ? value.bind(target) : value;+ },+ });++ const crawler = new BasicCrawler({+ requestQueue: observingManager as unknown as RequestQueue,+ requestHandler: async () => {},+ });++ await crawler.run();++ expect(fetches).toBeGreaterThan(0);+ expect(fetchedInsideTransaction).toBe(false);+ });++ test('onSkippedRequest bookkeeping survives an in-pipeline robots.txt skip', async () => {+ const crawler = new BasicCrawler({+ maxRequestRetries: 0,+ respectRobotsTxtFile: true,+ requestHandler: async () => {},+ onSkippedRequest: async ({ url, reason }) => {+ await (await KeyValueStore.open()).setValue('skipped', { url, reason });+ },+ });++ // Let the request into the crawler's queue, then disallow it during processing, so the skip+ // happens inside the request's transaction scope.+ const queue = await crawler.getRequestQueue();+ await queue.addRequest({ url: `http://${HOSTNAME}:${port}/disallowed` });+ vitest.spyOn(crawler as any, 'isAllowedBasedOnRobotsTxtFile').mockResolvedValue(false);++ await crawler.run();++ await expect(KeyValueStore.getValue('skipped')).resolves.toMatchObject({ reason: 'robotsTxt' });+ });++ test('an unrecognized write policy is rejected instead of falling back to the default', () => {+ const make = (transactionalStorage: unknown) =>+ new BasicCrawler({ requestHandler: async () => {}, transactionalStorage } as any);++ expect(() => make({ requestQueue: 'defered' })).toThrow(/requestQueue/);+ expect(() => make({ dataset: 'deferred' })).toThrow(/dataset/);++ expect(() => make(true)).not.toThrow();+ expect(() => make(false)).not.toThrow();+ expect(() => make({})).not.toThrow();+ expect(() => make({ requestQueue: 'deferred' })).not.toThrow();+ expect(() => make({ requestQueue: 'writeThrough' })).not.toThrow();+ });+ });+describe('addRequests input validation', () => {test('should throw error when passed a non-iterable value', async () => {const crawler = new BasicCrawler({diff --git a/test/core/storages/storage_transaction.test.ts b/test/core/storages/storage_transaction.test.tsnew file mode 100644index 000000000..c63ab61b2--- /dev/null+++ b/test/core/storages/storage_transaction.test.ts@@ -0,0 +1,898 @@+import { Readable } from 'node:stream';++import {+ createStorageTransaction,+ Dataset,+ getRequestId,+ KeyValueStore,+ MemoryStorageBackend,+ Request,+ RequestQueue,+ serviceLocator,+ withDirectStorageAccess,+ withStorageTransaction,+} from '@crawlee/core';++import { storage as timeoutStorage } from '@apify/timeout';++beforeEach(() => {+ serviceLocator.setStorageBackend(new MemoryStorageBackend());+});++describe('StorageTransaction', () => {+ describe('state machine', () => {+ test('commit applies buffered writes, rollback discards them', async () => {+ const store = await KeyValueStore.open();++ const committed = createStorageTransaction();+ await committed.run(async () => store.setValue('committed', { a: 1 }));+ await committed.commit();+ committed.dispose();++ const rolledBack = createStorageTransaction();+ await rolledBack.run(async () => store.setValue('rolled-back', { a: 1 }));+ rolledBack.rollback();+ rolledBack.dispose();++ await expect(store.getValue('committed')).resolves.toEqual({ a: 1 });+ await expect(store.getValue('rolled-back')).resolves.toBeNull();+ expect(committed.state).toBe('committed');+ expect(rolledBack.state).toBe('rolledBack');+ });++ test('rollback after a successful commit is a silent no-op', async () => {+ const store = await KeyValueStore.open();++ const transaction = createStorageTransaction();+ await transaction.run(async () => store.setValue('key', { a: 1 }));+ await transaction.commit();++ expect(() => transaction.rollback()).not.toThrow();+ expect(transaction.state).toBe('committed');+ await expect(store.getValue('key')).resolves.toEqual({ a: 1 });++ transaction.dispose();+ });++ test('double commit does not double-flush', async () => {+ const dataset = await Dataset.open();+ const pushDataSpy = vitest.spyOn(dataset.backend, 'pushData');++ const transaction = createStorageTransaction();+ await transaction.run(async () => dataset.pushData({ a: 1 }));+ await transaction.commit();+ await transaction.commit();+ transaction.dispose();++ expect(pushDataSpy).toHaveBeenCalledTimes(1);+ });++ test('a commit that throws lands in `failed` and later writes pass through', async () => {+ const dataset = await Dataset.open();+ const store = await KeyValueStore.open();+ vitest.spyOn(dataset.backend, 'pushData').mockRejectedValueOnce(new Error('backend exploded'));++ const transaction = createStorageTransaction();+ await transaction.run(async () => dataset.pushData({ a: 1 }));++ await expect(transaction.commit()).rejects.toThrow('backend exploded');+ expect(transaction.state).toBe('failed');+ expect(transaction.isActive).toBe(false);++ // Writes made inside the (still installed) scope of a failed transaction go straight to+ // real storage - e.g. an error handler running after a failed commit must not lose output.+ await transaction.run(async () => store.setValue('from-error-handler', { ok: true }));+ await expect(store.getValue('from-error-handler')).resolves.toEqual({ ok: true });++ transaction.dispose();+ });++ test('commit succeeds even when the ambient cancellation context has already been aborted', async () => {+ const store = await KeyValueStore.open();++ const transaction = createStorageTransaction();+ await transaction.run(async () => store.setValue('key', { a: 1 }));++ const controller = new AbortController();+ controller.abort();++ // Simulates a request-handler timeout that fired before commit: `tryCancel()` in the replayed+ // frontend calls would throw if commit inherited the shared, already-aborted controller.+ await timeoutStorage.run({ cancelTask: controller }, async () => {+ await transaction.commit();+ });++ expect(transaction.state).toBe('committed');+ transaction.dispose();++ await expect(store.getValue('key')).resolves.toEqual({ a: 1 });+ });+ });++ describe('scoping helpers', () => {+ test('withStorageTransaction commits on success and rolls back on throw', async () => {+ const store = await KeyValueStore.open();++ await withStorageTransaction(async () => {+ await store.setValue('success', { a: 1 });+ });++ await expect(+ withStorageTransaction(async () => {+ await store.setValue('failure', { a: 1 });+ throw new Error('handler failed');+ }),+ ).rejects.toThrow('handler failed');++ await expect(store.getValue('success')).resolves.toEqual({ a: 1 });+ await expect(store.getValue('failure')).resolves.toBeNull();+ });++ test('nested withStorageTransaction reuses the outer transaction', async () => {+ const store = await KeyValueStore.open();++ await withStorageTransaction(async (outer) => {+ await withStorageTransaction(async (inner) => {+ expect(inner).toBe(outer);+ await store.setValue('key', { a: 1 });+ });++ // The inner scope must not have committed on its own.+ const backend = (store as any).backend;+ await expect(backend.getValue('key')).resolves.toBeUndefined();+ });++ await expect(store.getValue('key')).resolves.toEqual({ a: 1 });+ });++ test('withDirectStorageAccess writes are immediate and survive rollback', async () => {+ const store = await KeyValueStore.open();++ const transaction = createStorageTransaction();+ await transaction.run(async () => {+ await withDirectStorageAccess(async () => store.setValue('direct', { a: 1 }));+ await store.setValue('buffered', { a: 1 });+ });+ transaction.rollback();+ transaction.dispose();++ await expect(store.getValue('direct')).resolves.toEqual({ a: 1 });+ await expect(store.getValue('buffered')).resolves.toBeNull();+ });+ });++ describe('isolation', () => {+ test('concurrent transactions on one frontend do not see each other; last commit wins', async () => {+ const store = await KeyValueStore.open();++ const first = createStorageTransaction();+ const second = createStorageTransaction();++ await first.run(async () => store.setValue('key', { from: 'first' }));+ await second.run(async () => {+ await expect(store.getValue('key')).resolves.toBeNull();+ await store.setValue('key', { from: 'second' });+ });+ await first.run(async () => {+ await expect(store.getValue('key')).resolves.toEqual({ from: 'first' });+ });++ await first.commit();+ await second.commit();+ first.dispose();+ second.dispose();++ await expect(store.getValue('key')).resolves.toEqual({ from: 'second' });+ });++ test('the journal is released on dispose, in every terminal state', async () => {+ const dataset = await Dataset.open();+ const store = await KeyValueStore.open();+ const queue = await RequestQueue.open();++ const transaction = createStorageTransaction({ policy: { requestQueue: 'deferred' } });+ await transaction.run(async () => {+ await dataset.pushData({ a: 1 });+ await store.setValue('key', { a: 1 });+ await queue.addRequest({ url: 'https://example.com' });+ });++ expect(transaction.journal).toHaveLength(3);++ // `failed` is the terminal state that neither `rollback()` nor an `open` check reaches -+ // dispose must release the journaled snapshots regardless of the outcome.+ vitest.spyOn(dataset.backend, 'pushData').mockRejectedValueOnce(new Error('boom'));+ await expect(transaction.commit()).rejects.toThrow('boom');+ expect(transaction.state).toBe('failed');++ transaction.dispose();++ expect(transaction.journal).toHaveLength(0);+ });+ });+});++describe('Dataset in a transaction', () => {+ test('read-your-own-writes through every read path', async () => {+ const dataset = await Dataset.open();+ await dataset.pushData({ n: 0 }); // pre-existing real item++ await withStorageTransaction(async (transaction) => {+ await dataset.pushData([{ n: 1 }, { n: 2 }]);++ const expected = [{ n: 0 }, { n: 1 }, { n: 2 }];++ await expect(dataset.getData()).resolves.toMatchObject({ total: 3, items: expected });+ await expect(dataset.export()).resolves.toEqual(expected);+ await expect(dataset.values()).resolves.toEqual(expected);+ await expect(dataset.entries()).resolves.toEqual(expected.map((item, i) => [i, item]));+ await expect(dataset.map((item) => item.n)).resolves.toEqual([0, 1, 2]);++ const iterated: unknown[] = [];+ for await (const item of dataset) {+ iterated.push(item);+ }+ expect(iterated).toEqual(expected);++ const info = await dataset.getInfo();+ expect(info.itemCount).toBe(3);++ expect(transaction.datasetItems.map(({ item }) => item)).toEqual([{ n: 1 }, { n: 2 }]);++ transaction.rollback();+ });++ // Nothing but the pre-existing item survives the rollback.+ await expect(dataset.getData()).resolves.toMatchObject({ total: 1, items: [{ n: 0 }] });+ });++ test('offset/limit/desc window across the real/buffered boundary', async () => {+ const dataset = await Dataset.open();+ await dataset.pushData([{ n: 0 }, { n: 1 }]);++ await withStorageTransaction(async (transaction) => {+ await dataset.pushData([{ n: 2 }, { n: 3 }]);++ await expect(dataset.getData({ offset: 1, limit: 2 })).resolves.toMatchObject({+ items: [{ n: 1 }, { n: 2 }],+ total: 4,+ });+ await expect(dataset.getData({ offset: 3, limit: 2 })).resolves.toMatchObject({ items: [{ n: 3 }] });+ await expect(dataset.getData({ desc: true, limit: 3 })).resolves.toMatchObject({+ items: [{ n: 3 }, { n: 2 }, { n: 1 }],+ total: 4,+ });++ transaction.rollback();+ });+ });++ test('a backend returning short pages does not pull buffered items into the real window', async () => {+ const dataset = await Dataset.open<{ n: number }>();+ await dataset.pushData(Array.from({ length: 12 }, (_, n) => ({ n })));++ // A backend honouring `skipEmpty` / `clean` / `unwind` returns fewer items than asked for while+ // the real items are far from exhausted - `total` stays honest, the page is just filtered.+ const realBackend = dataset.backend;+ dataset.backend = {+ getMetadata: async () => realBackend.getMetadata(),+ drop: async () => realBackend.drop(),+ purge: async () => realBackend.purge(),+ pushData: async (items) => realBackend.pushData(items),+ getData: async (options) => {+ const page = await realBackend.getData(options);+ const items = page.items.filter(({ n }) => ![1, 3, 5].includes(n));+ return { ...page, items, count: items.length };+ },+ };++ await withStorageTransaction(async (transaction) => {+ await dataset.pushData([{ n: 100 }, { n: 101 }]);++ // The window lies entirely within the 12 real items, so it must not reach the buffer at all -+ // deriving the buffered slice from the page's shortfall would append 100 and 101 here.+ await expect(dataset.getData({ offset: 0, limit: 5 })).resolves.toMatchObject({+ items: [{ n: 0 }, { n: 2 }, { n: 4 }],+ total: 14,+ });++ // ...and the buffer is reached exactly once, where the window really does cross into it.+ await expect(dataset.getData({ offset: 10, limit: 5 })).resolves.toMatchObject({+ items: [{ n: 10 }, { n: 11 }, { n: 100 }, { n: 101 }],+ total: 14,+ });++ transaction.rollback();+ });+ });++ test('values are captured at write time with full fidelity', async () => {+ const dataset = await Dataset.open();+ const pushDataSpy = vitest.spyOn(dataset.backend, 'pushData');++ const when = new Date('2023-01-01T00:00:00Z');+ const item: any = { when, missing: undefined, tags: new Set(['a']), n: 1 };++ await withStorageTransaction(async () => {+ await dataset.pushData(item);+ item.n = 2; // a later mutation must affect neither the read nor the commit++ const { items } = await dataset.getData();+ expect(items[0].n).toBe(1);+ });++ expect(pushDataSpy).toHaveBeenCalledTimes(1);+ const [committedItems] = pushDataSpy.mock.calls[0];+ expect(committedItems[0].when).toEqual(when);+ expect(committedItems[0].when).toBeInstanceOf(Date);+ expect(committedItems[0].tags).toEqual(new Set(['a']));+ expect('missing' in committedItems[0]).toBe(true);+ expect(committedItems[0].n).toBe(1);+ });++ test('items from multiple pushData calls are committed in order, in a single backend call', async () => {+ const dataset = await Dataset.open();+ const pushDataSpy = vitest.spyOn(dataset.backend, 'pushData');++ await withStorageTransaction(async () => {+ await dataset.pushData({ n: 1 });+ await dataset.pushData([{ n: 2 }, { n: 3 }]);+ });++ expect(pushDataSpy).toHaveBeenCalledTimes(1);+ expect(pushDataSpy).toHaveBeenCalledWith([{ n: 1 }, { n: 2 }, { n: 3 }]);+ });++ test('drop is rejected inside a transaction and allowed under withDirectStorageAccess', async () => {+ const dataset = await Dataset.open();++ await withStorageTransaction(async () => {+ await expect(dataset.drop()).rejects.toThrow(/cannot be used inside a storage transaction/);+ await withDirectStorageAccess(async () => dataset.drop());+ });+ });++ test('write stats count backend hits, not frontend calls', async () => {+ const dataset = await Dataset.open();++ await withStorageTransaction(async (transaction) => {+ await dataset.pushData({ n: 1 });+ transaction.rollback();+ });+ expect(dataset.stats.writeCount).toBe(0);++ await withStorageTransaction(async () => {+ await dataset.pushData({ n: 1 });+ await dataset.pushData({ n: 2 });+ });+ expect(dataset.stats.writeCount).toBe(1); // one collapsed backend call at commit+ });++ test('a merged read counts exactly one backend hit', async () => {+ const dataset = await Dataset.open();+ await dataset.pushData([{ n: 0 }, { n: 1 }]);++ await withStorageTransaction(async (transaction) => {+ await dataset.pushData({ n: 2 });++ const before = dataset.stats.readCount;+ await dataset.getData(); // real page + buffered items+ await dataset.getData({ desc: true, limit: 1 }); // buffer-only window -> getMetadata+ // Each merged read hits the backend once, counted once - not per branch, not per item.+ expect(dataset.stats.readCount - before).toBe(2);++ transaction.rollback();+ });+ });+});++describe('KeyValueStore in a transaction', () => {+ test('read-your-own-writes and collection length invariants', async () => {+ const store = await KeyValueStore.open();+ await store.setValue('real', { real: true });++ await withStorageTransaction(async (transaction) => {+ await store.setValue('buffered', { fresh: true });++ await expect(store.getValue('buffered')).resolves.toEqual({ fresh: true });+ await expect(store.recordExists('buffered')).resolves.toBe(true);++ const record = await store.getRecord('buffered');+ expect(record?.contentType).toMatch('application/json');+ expect(JSON.parse(record!.value.toString())).toEqual({ fresh: true });++ const keys = await store.keys();+ const values = await store.values();+ const entries = await store.entries();++ expect(new Set(keys)).toEqual(new Set(['real', 'buffered']));+ // A listing-only fix would pass `keys()` but lose buffered keys in `values()` / `entries()`.+ expect(values).toHaveLength(keys.length);+ expect(entries).toHaveLength(keys.length);+ expect(Object.fromEntries(entries)).toEqual({ real: { real: true }, buffered: { fresh: true } });++ const iterated: string[] = [];+ await store.forEachKey(async (key) => void iterated.push(key));+ expect(new Set(iterated)).toEqual(new Set(['real', 'buffered']));++ expect(transaction.keyValueStoreChanges[store.id]).toMatchObject({+ buffered: { changedValue: { fresh: true } },+ });++ transaction.rollback();+ });++ await expect(store.getValue('buffered')).resolves.toBeNull();+ await expect(store.getValue('real')).resolves.toEqual({ real: true });+ });++ test('listing reduces the journal a bounded number of times, not once per key', async () => {+ const store = await KeyValueStore.open();+ for (let i = 0; i < 20; i++) {+ await store.setValue(`real-${i}`, { i });+ }++ await withStorageTransaction(async (transaction) => {+ for (let i = 0; i < 20; i++) {+ await store.setValue(`buffered-${i}`, { i });+ }++ // `bufferedJournalEntries()` is the O(journal) reduction. Reading 40 keys' values must not+ // call it 40 times - the listing path builds one map and threads it through every per-key+ // read. A regression to per-key derivation makes this scale with the key count.+ const reduceSpy = vitest.spyOn(store as any, 'bufferedJournalEntries');++ const values = await store.values();+ expect(values).toHaveLength(40);++ // One reduction for the page listing, one shared across every record read: two, not forty.+ // The lower bound matters too - zero calls would mean the buffered reads were skipped entirely.+ expect(reduceSpy.mock.calls.length).toBeGreaterThanOrEqual(1);+ expect(reduceSpy.mock.calls.length).toBeLessThanOrEqual(2);++ reduceSpy.mockRestore();+ transaction.rollback();+ });+ });++ test('deletion is a first-class tombstone', async () => {+ const store = await KeyValueStore.open();+ await store.setValue('doomed', { alive: true });++ await withStorageTransaction(async (transaction) => {+ await store.setValue('doomed', null);++ await expect(store.getValue('doomed')).resolves.toBeNull();+ await expect(store.recordExists('doomed')).resolves.toBe(false);+ await expect(store.keys()).resolves.not.toContain('doomed');++ transaction.rollback();+ });++ // The rollback discarded the tombstone.+ await expect(store.getValue('doomed')).resolves.toEqual({ alive: true });++ await withStorageTransaction(async () => {+ await store.setValue('doomed', null);+ });++ // The committed tombstone deleted the record for real.+ await expect(store.getValue('doomed')).resolves.toBeNull();+ });++ test('prefix filtering applies to buffered keys too', async () => {+ const store = await KeyValueStore.open();+ await store.setValue('match-real', 1);++ await withStorageTransaction(async (transaction) => {+ await store.setValue('match-buffered', 2);+ await store.setValue('other', 3);++ const keys = await store.keys({ prefix: 'match-' });+ expect(new Set(keys)).toEqual(new Set(['match-real', 'match-buffered']));++ transaction.rollback();+ });+ });++ test('the last write per key wins at commit', async () => {+ const store = await KeyValueStore.open();+ const backend = (store as any).backend;+ const setValueSpy = vitest.spyOn(backend, 'setValue');++ await withStorageTransaction(async () => {+ await store.setValue('key', { version: 1 });+ await store.setValue('key', { version: 2 });+ });++ expect(setValueSpy).toHaveBeenCalledTimes(1);+ await expect(store.getValue('key')).resolves.toEqual({ version: 2 });+ });++ test('stream values are rejected inside a transaction, allowed with direct access', async () => {+ const store = await KeyValueStore.open();++ await withStorageTransaction(async () => {+ await expect(+ store.setValue('stream', Readable.from(['data']), { contentType: 'text/plain' }),+ ).rejects.toThrow(/stream/);++ await withDirectStorageAccess(async () =>+ store.setValue('stream', Readable.from([Buffer.from('data')]), { contentType: 'text/plain' }),+ );+ });++ await expect(store.getValue('stream')).resolves.toBe('data');+ });++ test('a stream value is rejected before the auto-saved cache is touched', async () => {+ const store = await KeyValueStore.open();+ // Warm the shared cache first - the cache update only runs for a key it already holds.+ const state = await store.getAutoSavedValue<{ a: number }>('STREAM_STATE', { a: 1 });++ await withStorageTransaction(async () => {+ await expect(+ store.setValue('STREAM_STATE', Readable.from(['data']), { contentType: 'text/plain' }),+ ).rejects.toThrow(/stream/);+ });++ // A stream is `typeof 'object'`, so a cache update reached before the rejection would splice the+ // stream's internals into the live state object every handler shares.+ expect(state).toEqual({ a: 1 });+ });++ test('auto-saved values bypass the transaction', async () => {+ const store = await KeyValueStore.open();++ await withStorageTransaction(async (transaction) => {+ // A buffered write to the state key must not leak into the shared auto-saved cache.+ await store.setValue('STATE', { poisoned: true });+ const state = await store.getAutoSavedValue<{ poisoned?: boolean; counter?: number }>('STATE', {});+ expect(state.poisoned).toBeUndefined();++ state.counter = 1;+ transaction.rollback();+ });++ // Mutations of the auto-saved object are deliberately not rolled back.+ const state = await store.getAutoSavedValue<{ counter?: number }>('STATE', {});+ expect(state.counter).toBe(1);+ });++ test('a rolled-back setValue does not leak into a warm auto-saved state cache', async () => {+ const store = await KeyValueStore.open();++ // Seed the shared cache *first* - a buffered write only reaches it when it is already warm.+ const state = await store.getAutoSavedValue<{ a: number; injected?: boolean }>('WARM_STATE', { a: 1 });++ const transaction = createStorageTransaction();+ await transaction.run(async () => store.setValue('WARM_STATE', { a: 2, injected: true }));+ transaction.rollback();+ transaction.dispose();++ // The live object every handler shares must be untouched, or `persistState` would write it out.+ expect(state).toEqual({ a: 1 });+ await expect(store.getAutoSavedValue('WARM_STATE', {})).resolves.toEqual({ a: 1 });+ await expect(store.getValue('WARM_STATE')).resolves.toBeNull();+ });++ test('a committed setValue does reach the shared auto-saved state object', async () => {+ const store = await KeyValueStore.open();+ const state = await store.getAutoSavedValue<{ a: number }>('COMMITTED_STATE', { a: 1 });++ await withStorageTransaction(async () => store.setValue('COMMITTED_STATE', { a: 2 }));++ // Identity is preserved, so handlers holding the reference observe the committed value.+ expect(state).toEqual({ a: 2 });+ await expect(store.getValue('COMMITTED_STATE')).resolves.toEqual({ a: 2 });+ });++ test('drop and clearCache are rejected inside a transaction', async () => {+ const store = await KeyValueStore.open();++ await withStorageTransaction(async () => {+ await expect(store.drop()).rejects.toThrow(/cannot be used inside a storage transaction/);+ expect(() => store.clearCache()).toThrow(/cannot be used inside a storage transaction/);+ });+ });+});++describe('RequestQueue in a transaction', () => {+ test('write-through (default): adds are applied immediately, journaled, and never rolled back', async () => {+ const queue = await RequestQueue.open();++ await withStorageTransaction(async (transaction) => {+ await queue.addRequests([{ url: 'https://example.com/a' }, { url: 'https://example.com/b' }]);++ // Applied immediately - visible to the backend (and thus to concurrent handlers).+ await expect(queue.backend.getRequest('https://example.com/a')).resolves.toBeDefined();++ // ...but still journaled for introspection.+ expect(transaction.enqueuedUrls.map(({ url }) => url)).toEqual([+ 'https://example.com/a',+ 'https://example.com/b',+ ]);++ transaction.rollback();+ });++ await expect(queue.getTotalCount()).resolves.toBe(2);+ });++ test('deferred: adds are buffered, readable within the transaction, and applied at commit', async () => {+ const queue = await RequestQueue.open();+ const addBatchSpy = vitest.spyOn(queue.backend, 'addBatchOfRequests');++ await withStorageTransaction(+ async () => {+ const info = await queue.addRequest({ url: 'https://example.com/a' });++ expect(info.wasAlreadyPresent).toBe(false);+ // The requestId is the locally derived hash, documented provisional.+ expect(info.requestId).toBe(getRequestId('https://example.com/a'));++ // Read-your-own-writes within the transaction...+ await expect(queue.getRequest('https://example.com/a')).resolves.toMatchObject({+ url: 'https://example.com/a',+ });+ await expect(queue.isEmpty()).resolves.toBe(false);+ const queueInfo = await queue.getInfo();+ expect(queueInfo.pendingRequestCount).toBe(1);++ // ...but nothing has reached the backend yet.+ expect(addBatchSpy).not.toHaveBeenCalled();++ // Intra-transaction dedup consults the read index.+ const duplicate = await queue.addRequest({ url: 'https://example.com/a' });+ expect(duplicate.wasAlreadyPresent).toBe(true);+ },+ { policy: { requestQueue: 'deferred' } },+ );++ // The commit replay carries no request ids - the backend assigns them.+ expect(addBatchSpy).toHaveBeenCalledTimes(1);+ const [committedRequests] = addBatchSpy.mock.calls[0];+ expect(committedRequests).toHaveLength(1);+ expect(committedRequests[0].id).toBeUndefined();++ await expect(queue.getTotalCount()).resolves.toBe(1);+ });++ test('deferred: a re-enqueued Request instance does not leak its id or handledAt into the commit', async () => {+ const queue = await RequestQueue.open();+ const addBatchSpy = vitest.spyOn(queue.backend, 'addBatchOfRequests');++ // Looks like it came from a backend: foreign id, already handled.+ const reenqueued = new Request({+ url: 'https://example.com/old',+ id: 'foreign-backend-id',+ handledAt: new Date().toISOString(),+ });++ await withStorageTransaction(+ async () => {+ const info = await queue.addRequests([reenqueued]);+ expect(info.processedRequests[0].wasAlreadyPresent).toBe(false);+ },+ { policy: { requestQueue: 'deferred' } },+ );++ // The contract under test is what crosses the backend boundary at commit: the stale id must not+ // reach the storage backend (a validating backend would reject it, failing the whole commit),+ // nor the handled marker (the request would be buried as already-handled).+ expect(addBatchSpy).toHaveBeenCalledTimes(1);+ const [committedRequests] = addBatchSpy.mock.calls[0];+ expect(committedRequests).toHaveLength(1);+ expect(committedRequests[0].id).toBeUndefined();+ expect(committedRequests[0].handledAt).toBeUndefined();++ // Which id the request ends up with is deliberately *not* asserted - that is the backend's own+ // affair: it deduplicates on `uniqueKey` and assigns the id itself (for the bundled backends,+ // the same `getRequestId(uniqueKey)` hash the provisional id was derived from, so the stored+ // request lands with the very id the journal time promised). Either way, it lands pending and+ // fetchable.+ await expect(queue.getInfo()).resolves.toMatchObject({ totalRequestCount: 1, pendingRequestCount: 1 });+ });++ test('deferred: requests the backend rejects at commit are warned about and skipped', async () => {+ const queue = await RequestQueue.open();+ const warningSpy = vitest.spyOn((queue as any).log, 'warning');++ // The backend refuses the request outright. Transient failures are the backend's own job to+ // retry, so what it reports as unprocessed is a semantic rejection - the commit warns and+ // moves on instead of dropping the link silently (or failing the whole request over it).+ const addBatchSpy = vitest.spyOn(queue.backend, 'addBatchOfRequests').mockImplementation(async (batch) => ({+ processedRequests: [],+ unprocessedRequests: batch.map((r) => ({ uniqueKey: r.uniqueKey, url: r.url })),+ }));++ await withStorageTransaction(+ async () => {+ await queue.addRequests([{ url: 'https://example.com/rejected' }]);+ },+ { policy: { requestQueue: 'deferred' } },+ );++ expect(addBatchSpy).toHaveBeenCalledTimes(1);+ expect(warningSpy).toHaveBeenCalledTimes(1);+ expect(warningSpy.mock.calls[0][0]).toMatch(/rejected by the request queue while committing/);+ await expect(queue.getTotalCount()).resolves.toBe(0); // the rejected request never landed+ });++ test('deferred: a rolled-back transaction leaves the dedup caches untouched', async () => {+ const queue = await RequestQueue.open();++ const transaction = createStorageTransaction({ policy: { requestQueue: 'deferred' } });+ const request = { url: 'https://example.com/a' };++ await transaction.run(async () => queue.addRequest({ ...request }));+ transaction.rollback();+ transaction.dispose();++ await expect(queue.getTotalCount()).resolves.toBe(0);++ // A later, genuine add of the same uniqueKey must still reach the backend.+ const addBatchSpy = vitest.spyOn(queue.backend, 'addBatchOfRequests');+ const info = await queue.addRequest({ ...request });+ expect(info.wasAlreadyPresent).toBe(false);+ expect(addBatchSpy).toHaveBeenCalledTimes(1);+ });++ test('deferred: requests already known to the dedup caches are not probed against the backend', async () => {+ const queue = await RequestQueue.open();+ const known = { url: 'https://example.com/known' };++ // Populates `requestCache` / `requestSeenCache` with the real backend id.+ const { requestId } = await queue.addRequest({ ...known });++ const getRequestSpy = vitest.spyOn(queue.backend, 'getRequest');++ await withStorageTransaction(+ async () => {+ const info = await queue.addRequest({ ...known });+ expect(info.wasAlreadyPresent).toBe(true);+ // The cached id is the real one the backend assigned, not the provisional hash.+ expect(info.requestId).toBe(requestId);++ // A genuinely unknown request still falls through to the probe.+ await queue.addRequest({ url: 'https://example.com/unknown' });+ },+ { policy: { requestQueue: 'deferred' } },+ );++ expect(getRequestSpy.mock.calls.map(([uniqueKey]) => uniqueKey)).toEqual(['https://example.com/unknown']);+ });++ test('write-through: an add still in flight when the transaction closes does not throw', async () => {+ const queue = await RequestQueue.open();+ const transaction = createStorageTransaction();++ // Mimics a request handler timing out: `handleRequest` rolls the transaction back while the+ // handler's storage calls keep running, so the add resumes against a closed transaction.+ const addBatch = queue.backend.addBatchOfRequests.bind(queue.backend);+ vitest.spyOn(queue.backend, 'addBatchOfRequests').mockImplementation(async (requests, options) => {+ transaction.rollback();+ return addBatch(requests, options);+ });++ await transaction.run(async () => {+ await expect(queue.addRequest({ url: 'https://example.com/a' })).resolves.toMatchObject({+ wasAlreadyPresent: false,+ });+ });++ // The entry is introspection-only and the transaction is already decided, so it is simply skipped.+ expect(transaction.enqueuedUrls).toEqual([]);+ transaction.dispose();++ // Write-through additions were never going to be rolled back.+ await expect(queue.getTotalCount()).resolves.toBe(1);+ });++ test('deferred: an add still in flight when the transaction closes reaches the real queue', async () => {+ const queue = await RequestQueue.open();+ const transaction = createStorageTransaction({ policy: { requestQueue: 'deferred' } });++ const getRequest = queue.backend.getRequest.bind(queue.backend);+ vitest.spyOn(queue.backend, 'getRequest').mockImplementation(async (uniqueKey) => {+ transaction.rollback();+ return getRequest(uniqueKey);+ });++ await transaction.run(async () => {+ await expect(queue.addRequest({ url: 'https://example.com/a' })).resolves.toMatchObject({+ wasAlreadyPresent: false,+ });+ });+ transaction.dispose();++ // Here the journal entry *is* the write, so it passes through rather than being dropped -+ // an unwanted (but deduplicated) addition beats losing a request silently.+ await expect(queue.getTotalCount()).resolves.toBe(1);+ });++ test('deferred: a multi-chunk addRequestsBatched call is fully applied when commit resolves', async () => {+ const queue = await RequestQueue.open();+ const requests = Array.from({ length: 1500 }, (_, i) => ({ url: `https://example.com/${i}` }));++ await withStorageTransaction(+ async () => {+ await queue.addRequestsBatched(requests);+ const { pendingRequestCount } = await queue.getInfo();+ expect(pendingRequestCount).toBe(1500);+ },+ { policy: { requestQueue: 'deferred' } },+ );++ // Replaying through the batched frontend wrapper instead of the backend call would resolve+ // with chunks still in flight and drop the tail.+ await expect(queue.getTotalCount()).resolves.toBe(1500);+ });++ test('requestsFromUrl sources are observed through their expanded URLs in `enqueuedUrls`', async () => {+ // Mocked at the HTTP client level: the list download goes through the queue's configured client.+ const lists: Record<string, string[]> = {+ 'http://example.com/list-1': ['https://example.com/a', 'https://example.com/b'],+ 'http://example.com/list-2': ['https://example.com/c'],+ };+ const httpClient = vitest.mockObject({+ async sendRequest(_request: any, _options?: any) {+ return new Response();+ },+ async stream() {+ return new Response();+ },+ });+ httpClient.sendRequest.mockImplementation(async (request: { url: string }) => {+ return new Response(lists[request.url]?.join('\n') ?? '');+ });++ // There is no `enqueuedUrlLists` on the view - the list is downloaded at call time and the+ // *fetched* URLs are journaled, under either policy.+ const queue = await RequestQueue.open(null, { httpClient });++ await withStorageTransaction(async (transaction) => {+ await queue.addRequests([{ requestsFromUrl: 'http://example.com/list-1', label: 'from-list' }]);++ // writeThrough (default): applied immediately, and journaled for introspection.+ expect(transaction.enqueuedUrls).toEqual([+ { url: 'https://example.com/a', label: 'from-list' },+ { url: 'https://example.com/b', label: 'from-list' },+ ]);+ });++ await expect(queue.getTotalCount()).resolves.toBe(2);++ await withStorageTransaction(+ async (transaction) => {+ await queue.addRequests([{ requestsFromUrl: 'http://example.com/list-2' }]);++ // deferred: journaled the same way, applied only at commit.+ expect(transaction.enqueuedUrls).toEqual([{ url: 'https://example.com/c' }]);+ },+ { policy: { requestQueue: 'deferred' } },+ );++ expect(httpClient.sendRequest).toHaveBeenCalledTimes(2);+ await expect(queue.getTotalCount()).resolves.toBe(3);+ });++ test('crawler bookkeeping operations are rejected inside a transaction', async () => {+ const queue = await RequestQueue.open();+ await queue.addRequest({ url: 'https://example.com' });+ const fetched = await withDirectStorageAccess(async () => queue.fetchNextRequest());++ await withStorageTransaction(async () => {+ await expect(queue.fetchNextRequest()).rejects.toThrow(/cannot be used inside a storage transaction/);+ await expect(queue.markRequestAsHandled(fetched!)).rejects.toThrow(+ /cannot be used inside a storage transaction/,+ );+ await expect(queue.reclaimRequest(fetched!)).rejects.toThrow(/cannot be used inside a storage transaction/);+ await expect(queue.drop()).rejects.toThrow(/cannot be used inside a storage transaction/);+ await expect(queue.purge()).rejects.toThrow(/cannot be used inside a storage transaction/);++ // The escape hatch still works for callers who genuinely mean it.+ await withDirectStorageAccess(async () => queue.reclaimRequest(fetched!));+ });+ });+});