Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
fix: Track modifications to crawlingContext.request in adaptive crawler (#4296)
Jan Buchar6c336ab5c827
diff --git a/docs/guides/adaptive-crawling/adaptive-crawling.mdx b/docs/guides/adaptive-crawling/adaptive-crawling.mdxindex 18ffa6664..d06d4cd8e 100644--- a/docs/guides/adaptive-crawling/adaptive-crawling.mdx+++ b/docs/guides/adaptive-crawling/adaptive-crawling.mdx@@ -49,7 +49,11 @@ flowchart TD- **Predicted client-only:** the handler runs in the browser.- **Detection:** on some requests, the handler runs in the browser *and* over HTTP. The browser result is committed, the two are compared with the [`resultComparator`](#comparing-results-during-detection), and the verdict trains the predictor.-Each attempt runs in its own storage transaction (see [Result storage](./result-storage)). The writes of an attempt that loses — a failed HTTP run, or the HTTP run of a detection — are rolled back, and its log messages are dropped with it. The HTTP run of a detection also sees `useState()` as it was before the browser run, so both runs start from the same state.+Each attempt runs in its own storage transaction (see [Result storage](./result-storage)). The writes of an attempt that loses — a failed HTTP run, or the HTTP run of a detection — are rolled back, and its log messages are dropped with it.++The same goes for the `request` object and the `useState()` state, which are not covered by the transaction: the crawler records the changes an attempt makes to them and undoes those of a losing attempt. A browser run that fails keeps its changes, so `userData` stashed for the retry survives, as it does in the other crawlers. The HTTP run of a detection works on copies of both, as they were before the browser run, so both runs start from the same conditions.++The state is shared with concurrently running handlers. If one of them wrote the same property while the losing attempt was running, the newer value is kept rather than overwritten.:::caution The request handler may run twicediff --git a/docs/guides/result-storage.mdx b/docs/guides/result-storage.mdxindex 865ce61d9..07bb947f6 100644--- a/docs/guides/result-storage.mdx+++ b/docs/guides/result-storage.mdx@@ -154,7 +154,7 @@ This is **on by default** for every crawler and covers the whole request lifecycUnder the `deferred` policy, the `requestId` returned by an `addRequests()` call inside a handler is provisional — the real id is assigned by the storage backend when the transaction commits.-- **Shared state.** The object returned by `useState()` / <ApiLink to="core/class/KeyValueStore#getAutoSavedValue">`KeyValueStore.getAutoSavedValue()`</ApiLink> is the sanctioned live channel shared by all handlers, and it stays live: mutations of it are **not** rolled back when a handler fails.+- **Shared state.** The object returned by `useState()` / <ApiLink to="core/class/KeyValueStore#getAutoSavedValue">`KeyValueStore.getAutoSavedValue()`</ApiLink> is deliberately exempt. It is the live channel shared by all handlers: a write is visible to every other handler the moment it is made, and a counter incremented from several handlers at once stays correct only because of that. Buffering it per request would hide those writes until commit and lose concurrent updates, so mutations of it are **not** rolled back when a handler fails — a failed handler that incremented a counter leaves the increment behind. (<ApiLink to="playwright-crawler/class/AdaptivePlaywrightCrawler">`AdaptivePlaywrightCrawler`</ApiLink> undoes the state writes of a request handler attempt it discards; see [Adaptive crawling](./adaptive-crawling#how-it-works).)- **Cross-storage atomicity.** The commit spans multiple storages and multiple calls; delivery is *at-least-once*. If a commit fails partway, the request fails and is retried, and the retry may re-apply what already landed.diff --git a/docs/public-api/crawlee-playwright.api.md b/docs/public-api/crawlee-playwright.api.mdindex c9151dcdb..112e67216 100644--- a/docs/public-api/crawlee-playwright.api.md+++ b/docs/public-api/crawlee-playwright.api.md@@ -24,7 +24,7 @@ import { CheerioAPI } from 'cheerio';import { Configuration } from '@crawlee/browser';import type { CrawlerRemoteBrowserOptions } from '@crawlee/browser';import type { CrawlingContext } from '@crawlee/basic';-import type { CrawlingRequest } from '@crawlee/browser';+import { CrawlingRequest } from '@crawlee/browser';import type { Dictionary } from '@crawlee/types';import type { Download } from 'playwright';import type { EnqueueLinksOptions } from '@crawlee/basic';diff --git a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.tsindex 44fd519c9..15003f081 100644--- a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts+++ b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts@@ -1,11 +1,4 @@-import type {- BrowserHook,- LoadedRequest,- CrawlingRequest,- RouterHandler,- RouteSchemas,- RoutesFromSchemas,-} from '@crawlee/browser';+import type { BrowserHook, LoadedRequest, RouterHandler, RouteSchemas, RoutesFromSchemas } from '@crawlee/browser';import { setTimeout as delay } from 'node:timers/promises';import { isDeepStrictEqual } from 'node:util';@@ -29,7 +22,7 @@ import {Router,Statistics,} from '@crawlee/basic';-import { extractUrlsFromPage } from '@crawlee/browser';+import { CrawlingRequest, extractUrlsFromPage } from '@crawlee/browser';import type { CheerioCrawlingContext } from '@crawlee/cheerio';import { CheerioCrawler } from '@crawlee/cheerio';import { ResponseWithUrl } from '@crawlee/http-client';@@ -42,7 +35,7 @@ import type {} from '@crawlee/core';import { createStorageTransaction, EnqueueStrategy, OwnedOrInjected } from '@crawlee/core';import type { Dictionary, Awaitable } from '@crawlee/types';-import { extractUrlsFromCheerio, parseArgument } from '@crawlee/utils/internal';+import { extractUrlsFromCheerio, MutationJournal, parseArgument } from '@crawlee/utils/internal';import { type Cheerio, type CheerioAPI } from 'cheerio';import type { AnyNode } from 'domhandler';import type { Page } from 'playwright';@@ -63,9 +56,24 @@ import {RenderingTypePredictor,} from './utils/rendering-type-prediction.js';-type Result<TResult> =- | { result: TResult; ok: true; logs?: LogProxyCall[] }- | { error: unknown; ok: false; logs?: LogProxyCall[] };+/**+ * Everything one request handler attempt produced. Present whether or not the attempt succeeded - the+ * transaction and the context modifications of a failed attempt still need to be settled by the caller.+ */+type AttemptResult = {+ transaction: StorageTransaction;+ contextModifications: ContextModifications;+ logs: LogProxyCall[];+} & ({ ok: true } | { ok: false; error: unknown });++/**+ * The writes an attempt made to the shared `request` and `useState()` objects. Kept for an attempt whose+ * outcome stands (committed, or failed and handed to the retry logic), reverted for one that is discarded.+ */+interface ContextModifications {+ request: MutationJournal;+ state: MutationJournal;+}const adaptiveStatisticStateSchema = z.object({/** How many requests were handled by the HTTP-only request handler. */@@ -619,15 +627,19 @@ export class AdaptivePlaywrightCrawler</*** Runs one request handler attempt inside its own {@apilink StorageTransaction}, wrapping the inner* (static or browser) context pipeline. The transaction is pushed to `transactions` *at creation- * time, before the `try`* - the `ok: false` branch of the returned {@apilink Result} carries no- * result, and failed attempts are routine here. The caller owns the outcome and disposal.+ * time, before the `try`*, as failed attempts are routine here. The caller owns the outcome and+ * disposal, including closing or reverting the {@apilink ContextModifications}.+ *+ * `request` defaults to the context's; a detection rerun passes a copy so it sees the request as it+ * was before the browser attempt.*/async #crawlOne(renderingType: RenderingType,context: CrawlingContext,useStateFunction: (defaultValue?: Dictionary) => Promise<Dictionary>,transactions: StorageTransaction[],- ): Promise<Result<StorageTransaction>> {+ request: CrawlingContext['request'] = context.request,+ ): Promise<AttemptResult> {const transaction = createStorageTransaction({policy: this.#attemptWritePolicy,commitTimeoutMillis: this.internalTimeoutMillis,@@ -638,13 +650,20 @@ export class AdaptivePlaywrightCrawler<const deferredCleanup: (() => Promise<unknown>)[] = [];+ const contextModifications: ContextModifications = {+ request: new MutationJournal(),+ state: new MutationJournal(),+ };+const subCrawlerContext = Object.defineProperties({},Object.getOwnPropertyDescriptors(context),) as typeof context & { [navigationDeadlineKey]?: number };const attemptBoundContextHelpers = {- useState: useStateFunction,+ request: contextModifications.request.track(request),+ useState: async (defaultValue?: Dictionary) =>+ contextModifications.state.track(await useStateFunction(defaultValue)),log: this.#createLogProxy(context.log, logs),registerDeferredCleanup: (cleanup: () => Promise<unknown>) => deferredCleanup.push(cleanup),// The sub-crawler pipeline keeps this attempt's navigation deadline on `subCrawlerContext`, but the@@ -690,9 +709,9 @@ export class AdaptivePlaywrightCrawler<'Request handler timed out',);- return { result: transaction, ok: true, logs };+ return { ok: true, transaction, contextModifications, logs };} catch (error) {- return { error, ok: false, logs };+ return { ok: false, error, transaction, contextModifications, logs };} finally {await Promise.all(deferredCleanup.map((cleanup) => cleanup()));}@@ -730,10 +749,14 @@ export class AdaptivePlaywrightCrawler<transactions,);- if (plainHTTPRun.ok && this.#resultChecker(plainHTTPRun.result, { request: crawlingContext.request })) {+ if (+ plainHTTPRun.ok &&+ this.#resultChecker(plainHTTPRun.transaction, { request: crawlingContext.request })+ ) {crawlingContext.log.debug(`HTTP-only request handler succeeded for ${crawlingContext.request.url}`);- plainHTTPRun.logs?.forEach(([log, method, ...args]) => log[method](...(args as [any, any])));- await plainHTTPRun.result.commit();+ plainHTTPRun.logs.forEach(([log, method, ...args]) => log[method](...(args as [any, any])));+ this.#keepContextModifications(plainHTTPRun.contextModifications);+ await plainHTTPRun.transaction.commit();return;}@@ -749,6 +772,7 @@ export class AdaptivePlaywrightCrawler<actualError instanceof ContextPipelineInterruptedError ||(await this.#shouldPropagateError(actualError, crawlingContext as any))) {+ this.#keepContextModifications(plainHTTPRun.contextModifications);throw actualError;}@@ -763,47 +787,31 @@ export class AdaptivePlaywrightCrawler<this.statistics.state.renderingTypeMispredictions++;httpResultRejected = true;}++ // The browser attempt starts from the request and state as they were before the rejected one.+ this.#discardContextModifications(plainHTTPRun.contextModifications, crawlingContext);}crawlingContext.log.debug(`Running browser request handler for ${crawlingContext.request.url}`);this.statistics.state.browserRequestHandlerRuns++;- // Run the request handler in a browser. The copy of the crawler state is kept so that we can perform- // a rendering type detection if necessary. Without this measure, the HTTP request handler would run- // under different conditions, which could change its behavior. Changes done to the crawler state by- // the HTTP request handler will not be committed to the actual storage.- const stateTracker = {- stateCopy: null,- async getLiveState(defaultValue: Dictionary = {}) {- const state = await crawlingContext.useState(defaultValue);-- if (this.stateCopy === null) {- this.stateCopy = JSON.parse(JSON.stringify(state));- }-- return state;- },- async getStateCopy(defaultValue: Dictionary = {}) {- if (this.stateCopy === null) {- return defaultValue;- }- return this.stateCopy;- },- };-const browserRun = await this.#crawlOne('clientOnly',crawlingContext,- stateTracker.getLiveState.bind(stateTracker),+ crawlingContext.useState,transactions,);+ // A failed browser attempt is handed to the retry logic with its writes - `userData` stashed for the+ // next attempt must survive.+ this.#keepContextModifications(browserRun.contextModifications);+if (!browserRun.ok) {throw browserRun.error;}- browserRun.logs?.forEach(([log, method, ...args]) => log[method](...(args as [any, any])));- await browserRun.result.commit();+ browserRun.logs.forEach(([log, method, ...args]) => log[method](...(args as [any, any])));+ await browserRun.transaction.commit();// Past the shutdown fence, `teardown()` has already drained - and possibly torn down the predictor.if (httpResultRejected && !this.#shutDown) {@@ -814,12 +822,22 @@ export class AdaptivePlaywrightCrawler<const detectionPromise = (async () => {crawlingContext.log.debug(`Detecting rendering type for ${crawlingContext.request.url}`);// The detection attempt's transaction is never committed - its writes exist only for the- // result comparison.+ // result comparison. It runs on copies of the request and the crawler state with the browser+ // attempt's writes undone, so both attempts start from the same conditions, and the copies go+ // away with it.+ const { request: requestJournal, state: stateJournal } = browserRun.contextModifications;+ let stateCopy: Dictionary | undefined;const plainHTTPRun = await this.#crawlOne('static',crawlingContext,- stateTracker.getStateCopy.bind(stateTracker),+ async (defaultValue) =>+ (stateCopy ??= stateJournal.undoOnto(+ JSON.parse(JSON.stringify(await crawlingContext.useState(defaultValue))),+ )),transactions,+ requestJournal.undoOnto(+ CrawlingRequest.fromSchema(JSON.parse(JSON.stringify(crawlingContext.request))),+ ),);const detectionResult: RenderingType | undefined = (() => {@@ -827,9 +845,13 @@ export class AdaptivePlaywrightCrawler<return 'clientOnly';}- const comparisonResult = this.#resultComparator(plainHTTPRun.result, browserRun.result, {- request: crawlingContext.request,- });+ const comparisonResult = this.#resultComparator(+ plainHTTPRun.transaction,+ browserRun.transaction,+ {+ request: crawlingContext.request,+ },+ );if (comparisonResult === true || comparisonResult === 'equal') {return 'static';}@@ -861,6 +883,26 @@ export class AdaptivePlaywrightCrawler<}}+ #keepContextModifications(modifications: ContextModifications): void {+ modifications.request.close();+ modifications.state.close();+ }++ #discardContextModifications(modifications: ContextModifications, { request, log }: CrawlingContext): void {+ // Nothing else writes to a request while its handler runs, so a conflict means a write that bypassed+ // the context (e.g. through a stashed reference) - it stays, but it is not what the retry logic expects.+ modifications.request.revert({+ onConflict: (key) =>+ log.warning(`Request ${request.url}: '${String(key)}' was modified outside the request handler`),+ });+ // The state is shared with concurrent handlers; a conflict is one of them having written the same+ // property meanwhile. Their value wins, and this attempt's write stays with it.+ modifications.state.revert({+ onConflict: (key) =>+ log.debug(`Crawler state '${String(key)}' was changed concurrently; keeping the newer value`),+ });+ }+#storeRenderingType(request: CrawlingContext['request'], renderingType: RenderingType): void {// Deliberately not awaited: a predictor that persists asynchronously gets to keep batching its writes,// and `drainRenderingDetections()` catches whatever is still pending.diff --git a/packages/utils/src/internal.ts b/packages/utils/src/internal.tsindex c0370cf94..736a7100f 100644--- a/packages/utils/src/internal.ts+++ b/packages/utils/src/internal.ts@@ -4,6 +4,7 @@ export { extractUrlsFromCheerio } from './internals/cheerio.js';export { tryAbsoluteURL } from './internals/extract-urls.js';export { expandShadowRoots, URL_NO_COMMAS_REGEX, URL_WITH_COMMAS_REGEX } from './internals/general.js';export * from './internals/iterables.js';+export { MutationJournal } from './internals/mutation-journal.js';export * from './internals/url.js';export * from './internals/validation.js';export * as schemas from './internals/schemas.js';diff --git a/packages/utils/src/internals/mutation-journal.ts b/packages/utils/src/internals/mutation-journal.tsnew file mode 100644index 000000000..e63d2d57b--- /dev/null+++ b/packages/utils/src/internals/mutation-journal.ts@@ -0,0 +1,202 @@+interface JournalEntry {+ target: object;+ /** Location of `target` under the root it was reached from, for replaying onto another root. */+ path: PropertyKey[];+ key: PropertyKey;+ had: boolean;+ prev: unknown;+ next: unknown;+ deleted: boolean;+}++/**+ * Only plain objects and arrays are wrapped when reached through a tracked object. Built-ins such as `Map`+ * or `Set` reject a proxy receiver in their methods, and the tracked objects are JSON-shaped by contract+ * anyway - such values are handed out as they are, and their mutations are not recorded.+ */+const isTrackable = (value: unknown): value is object =>+ Array.isArray(value) ||+ (typeof value === 'object' &&+ value !== null &&+ (Object.getPrototypeOf(value) === Object.prototype || Object.getPrototypeOf(value) === null));++/** Every proxy any journal has handed out, keyed by the proxy. Shared, as a value read through one attempt's+ * proxy may be written back through another's. */+const proxyTargets = new WeakMap<object, object>();++/**+ * Replaces journal proxies in `value` with the objects behind them, in place for nested ones. A value read+ * through a proxy and written back (`state.x = state.x ?? {}`) would otherwise store the proxy itself, and+ * every later attempt would wrap it once more.+ */+function unwrap(value: unknown, seen = new Set<object>()): unknown {+ const target = typeof value === 'object' && value !== null ? proxyTargets.get(value) : undefined;+ if (target) return target;+ if (!isTrackable(value) || seen.has(value)) return value;++ seen.add(value);+ for (const key of Object.keys(value)) {+ const inner = (value as Record<string, unknown>)[key];+ const unwrapped = unwrap(inner, seen);+ if (unwrapped !== inner) (value as Record<string, unknown>)[key] = unwrapped;+ }+ return value;+}++/**+ * Records property writes made through the proxies it hands out, applying them to the real objects as+ * they happen, so that a losing attempt's writes can be undone afterwards - on the objects themselves+ * ({@apilink MutationJournal.revert|`revert()`}), or on a copy of them+ * ({@apilink MutationJournal.undoOnto|`undoOnto()`}) to reconstruct how they looked before the attempt.+ *+ * An undo on the live object only touches a property that still holds the value the attempt wrote - a+ * value written by someone else in the meantime is left alone and reported via `onConflict`. Writes that+ * bypass the proxy are invisible until they trip that guard.+ */+export class MutationJournal {+ readonly #entries: JournalEntry[] = [];+ readonly #proxies = new WeakMap<object, object>();+ readonly #paths = new WeakMap<object, PropertyKey[]>();+ #closed = false;++ track<T extends object>(target: T, path: PropertyKey[] = []): T {+ const cached = this.#proxies.get(target);+ if (cached) return cached as T;++ this.#paths.set(target, path);+ const entry = (key: PropertyKey, fields: Omit<JournalEntry, 'target' | 'path' | 'key'>) =>+ this.#entries.push({ target, path, key, ...fields });++ // Accessors run with `this` as the real object, not the proxy: class accessors may touch `#private`+ // members, which throw on a proxy receiver. An accessor write is therefore journaled as its own+ // entry and undone through the same setter, rather than through whatever it writes underneath.+ const proxy = new Proxy(target, {+ get: (object, key) => {+ const value = Reflect.get(object, key);+ return isTrackable(value) ? this.track(value, [...path, key]) : value;+ },+ set: (object, key, rawNext) => {+ const next = unwrap(rawNext);+ const had = Reflect.has(object, key);+ const prev = Reflect.get(object, key);+ this.#recordArrayTruncation(object, key, next);+ if (!Reflect.set(object, key, next)) return false;+ // An array's `length` is implied by its elements and never undone on its own.+ if (!this.#closed && !(Array.isArray(object) && key === 'length')) {+ entry(key, { had, prev, next, deleted: false });+ }+ return true;+ },+ deleteProperty: (object, key) => {+ const had = Reflect.has(object, key);+ const prev = Reflect.get(object, key);+ if (!Reflect.deleteProperty(object, key)) return false;+ if (!this.#closed && had) entry(key, { had, prev, next: undefined, deleted: true });+ return true;+ },+ });++ this.#proxies.set(target, proxy);+ proxyTargets.set(proxy, target);+ return proxy;+ }++ /**+ * Shrinking an array's `length` drops elements without a trap firing for them. They are recorded as+ * deletions, from the highest index down, so the reverse replay reinserts them from the lowest up.+ */+ #recordArrayTruncation(object: object, key: PropertyKey, next: unknown): void {+ if (this.#closed || !Array.isArray(object) || key !== 'length') return;+ const path = this.#paths.get(object)!;++ for (let i = object.length - 1; i >= (next as number); i--) {+ if (i in object) {+ this.#entries.push({+ target: object,+ path,+ key: String(i),+ had: true,+ prev: object[i],+ next: undefined,+ deleted: true,+ });+ }+ }+ }++ /**+ * Restores one entry on `target`. Array elements are handled positionally - an added element is removed+ * and a removed one reinserted, shifting the rest - so the result stays something a sequence of `push`+ * and `splice` calls could have produced, even next to elements another writer added meanwhile.+ */+ static #undo(target: object, { key, had, deleted }: JournalEntry, value: unknown): void {+ if (Array.isArray(target) && typeof key === 'string' && Number.isInteger(Number(key))) {+ if (deleted) {+ target.splice(Number(key), 0, value);+ } else if (had) {+ target[Number(key)] = value;+ } else {+ target.splice(Number(key), 1);+ }+ return;+ }++ if (had) {+ Reflect.set(target, key, value);+ } else {+ Reflect.deleteProperty(target, key);+ }+ }++ /** Stops recording. Proxies keep writing through. */+ close(): void {+ this.#closed = true;+ }++ /** Undoes the recorded writes, newest first, and stops recording. */+ revert({ onConflict }: { onConflict?: (key: PropertyKey) => void } = {}): void {+ this.close();++ for (let i = this.#entries.length - 1; i >= 0; i--) {+ const entry = this.#entries[i];+ const { target, key, next, deleted } = entry;+ const untouched = deleted ? !Reflect.has(target, key) : Reflect.get(target, key) === next;++ if (!untouched) {+ onConflict?.(key);+ continue;+ }++ MutationJournal.#undo(target, entry, entry.prev);+ }++ this.#entries.length = 0;+ }++ /**+ * Applies the inverse of the recorded writes onto `root`, a JSON copy of the tracked object, and returns+ * it. The journal is left as it is. Restored values are JSON-cloned too, so the copy shares nothing with+ * the original and non-JSON values drop out the same way on both.+ *+ * Entries are located by the path under which their object was *first* reached, so an object reachable+ * through two paths is undone under the first one only.+ */+ undoOnto<T extends object>(root: T): T {+ for (let i = this.#entries.length - 1; i >= 0; i--) {+ const entry = this.#entries[i];+ const target = entry.path.reduce<unknown>(+ (object, step) => (object as Record<PropertyKey, unknown>)?.[step],+ root,+ );+ if (typeof target !== 'object' || target === null) continue;++ MutationJournal.#undo(+ target,+ entry,+ isTrackable(entry.prev) ? JSON.parse(JSON.stringify(entry.prev)) : entry.prev,+ );+ }++ return root;+ }+}diff --git a/test/core/crawlers/adaptive-playwright-crawler.test.ts b/test/core/crawlers/adaptive-playwright-crawler.test.tsindex b76a0cc48..411f8957e 100644--- a/test/core/crawlers/adaptive-playwright-crawler.test.ts+++ b/test/core/crawlers/adaptive-playwright-crawler.test.ts@@ -138,6 +138,10 @@ describe('AdaptivePlaywrightCrawler', () => {res.setHeader('Location', 'https://google.com');res.send('Redirecting...');});++ app.get('/plain', (_req, res) => {+ res.status(200).type('text/plain').send('hello');+ });});afterAll(async () => {server.close();@@ -800,6 +804,144 @@ describe('AdaptivePlaywrightCrawler', () => {expect(await (await KeyValueStore.open()).getValue('CRAWLEE_STATE')).toEqual({ count: 3 });});+ describe('writes of a discarded attempt', () => {+ test('a rejected HTTP-only attempt does not affect retries of the browser attempt', async () => {+ // The HTTP crawler rejects the content type with `request.noRetry = true` before the handler runs.+ const renderingTypePredictor = makeRiggedRenderingTypePredictor({+ detectionProbabilityRecommendation: 0,+ renderingType: 'static',+ });++ const requestHandler = vi.fn(async () => {+ if (requestHandler.mock.calls.length === 1) throw new Error('a transient failure');+ });+ const failedRequestHandler = vi.fn();++ const crawler = await makeOneshotCrawler(+ { requestHandler, renderingTypePredictor, failedRequestHandler, maxRequestRetries: 1 },+ [`http://${HOSTNAME}:${port}/plain`],+ );+ await crawler.run();++ expect(requestHandler).toHaveBeenCalledTimes(2);+ expect(failedRequestHandler).not.toHaveBeenCalled();+ });++ test('request and state writes of a failed HTTP-only attempt are not visible to the browser attempt', async () => {+ const renderingTypePredictor = makeRiggedRenderingTypePredictor({+ detectionProbabilityRecommendation: 0,+ renderingType: 'static',+ });++ const seenUserData: Dictionary[] = [];+ const requestHandler: AdaptivePlaywrightCrawlerOptions['requestHandler'] = vi.fn(+ async ({ request, useState }) => {+ seenUserData.push({ ...request.userData });+ const state = await useState({ count: 0, visited: [] as string[] });+ state.count++;+ state.visited.push(request.url);+ request.userData.attempts = (request.userData.attempts ?? 0) + 1;+ request.label = 'touched';+ if (seenUserData.length === 1) throw new Error('a transient failure');+ },+ );++ const crawler = await makeOneshotCrawler({ requestHandler, renderingTypePredictor }, [+ `http://${HOSTNAME}:${port}/static`,+ ]);+ await crawler.run();++ expect(seenUserData).toEqual([{}, {}]);+ expect(await (await KeyValueStore.open()).getValue('CRAWLEE_STATE')).toEqual({+ count: 1,+ visited: [`http://${HOSTNAME}:${port}/static`],+ });+ });++ test('request writes made during a failed browser attempt are visible to the retry', async () => {+ const renderingTypePredictor = makeRiggedRenderingTypePredictor({+ detectionProbabilityRecommendation: 0,+ renderingType: 'clientOnly',+ });++ const seenAttempts: unknown[] = [];+ const requestHandler: AdaptivePlaywrightCrawlerOptions['requestHandler'] = vi.fn(async ({ request }) => {+ seenAttempts.push(request.userData.attempt);+ request.userData.attempt = seenAttempts.length;+ if (seenAttempts.length === 1) throw new Error('a transient failure');+ });++ const crawler = await makeOneshotCrawler({ requestHandler, renderingTypePredictor, maxRequestRetries: 1 }, [+ `http://${HOSTNAME}:${port}/static`,+ ]);+ await crawler.run();++ expect(seenAttempts).toEqual([undefined, 1]);+ });++ test('the detection rerun (HTTP) sees the live state in case the browser attempt never read it', async () => {+ const renderingTypePredictor = makeRiggedRenderingTypePredictor({+ detectionProbabilityRecommendation: 1,+ renderingType: 'clientOnly',+ });+ await (await KeyValueStore.open()).setValue('CRAWLEE_STATE', { seeded: true });++ const seen: Dictionary[] = [];+ const requestHandler: AdaptivePlaywrightCrawlerOptions['requestHandler'] = async (context) => {+ // `page` throws in the HTTP-only run; the browser run never touches the state.+ try {+ void context.page;+ return;+ } catch {}++ const state = await context.useState<Dictionary>({ fresh: true });+ seen.push({ ...state });+ state.rerun = true;+ seen.push({ ...(await context.useState({ fresh: true })) });+ };++ const crawler = await makeOneshotCrawler({ requestHandler, renderingTypePredictor }, [+ `http://${HOSTNAME}:${port}/static`,+ ]);+ await crawler.run();++ expect(seen).toEqual([{ seeded: true }, { seeded: true, rerun: true }]);+ expect(await (await KeyValueStore.open()).getValue('CRAWLEE_STATE')).toEqual({ seeded: true });+ });++ test('the detection rerun (HTTP) sees the request as it was before the browser attempt', async () => {+ const renderingTypePredictor = makeRiggedRenderingTypePredictor({+ detectionProbabilityRecommendation: 1,+ renderingType: 'clientOnly',+ });++ const seenUserData: Dictionary[] = [];+ let browserRequest: Request | undefined;+ const requestHandler: AdaptivePlaywrightCrawlerOptions['requestHandler'] = async (context) => {+ seenUserData.push({ ...context.request.userData });+ context.request.userData.attempts = (context.request.userData.attempts ?? 0) + 1;+ try {+ void context.page;+ browserRequest = context.request;+ } catch {}+ };++ const url = `http://${HOSTNAME}:${port}/static`;+ const crawler = await makeOneshotCrawler(+ {+ requestHandler,+ renderingTypePredictor,+ requestList: await RequestList.open({ sources: [{ url, userData: { seeded: true } }] }),+ },+ [],+ );+ await crawler.run();++ expect(seenUserData).toEqual([{ seeded: true }, { seeded: true }]);+ expect(browserRequest?.userData).toEqual({ seeded: true, attempts: 1 });+ });+ });+test('should return deeply equal but not identical state objects across handler runs', async () => {// Force detection to happenconst renderingTypePredictor = makeRiggedRenderingTypePredictor({diff --git a/test/utils/mutation-journal.test.ts b/test/utils/mutation-journal.test.tsnew file mode 100644index 000000000..7f7f84df0--- /dev/null+++ b/test/utils/mutation-journal.test.ts@@ -0,0 +1,147 @@+import { types } from 'node:util';++import { CrawlingRequest } from '@crawlee/basic';+import { MutationJournal } from '@crawlee/utils/internal';++describe('MutationJournal', () => {+ test('reverts nested writes, deletions and array pushes in reverse order', () => {+ const target = { count: 0, nested: { keep: 1, drop: 2 }, list: ['a'] };+ const journal = new MutationJournal();+ const proxy = journal.track(target);++ proxy.count++;+ proxy.nested.keep = 10;+ delete (proxy.nested as Partial<typeof target.nested>).drop;+ proxy.list.push('b', 'c');+ (proxy as Record<string, unknown>).added = true;++ journal.revert();++ expect(target).toEqual({ count: 0, nested: { keep: 1, drop: 2 }, list: ['a'] });+ });++ test('undoes array writes positionally, so concurrent pushes leave no holes', () => {+ const state = { visited: [] as string[] };+ const a = new MutationJournal();+ const b = new MutationJournal();++ a.track(state).visited.push('a');+ b.track(state).visited.push('b');+ a.revert();++ expect(state.visited).toEqual(['b']);+ });++ test('restores elements dropped by truncation, shift and splice', () => {+ const target = { list: ['a', 'b', 'c', 'd'] };+ const journal = new MutationJournal();+ const proxy = journal.track(target);++ proxy.list.shift();+ proxy.list.splice(1, 1);+ proxy.list.length = 1;++ expect(target.list).toEqual(['b']);+ journal.revert();++ expect(target.list).toEqual(['a', 'b', 'c', 'd']);+ });++ test('a value read through a proxy and written back is stored unwrapped', () => {+ const state = { seen: {} as Record<string, boolean>, items: [{ id: 1 }, { id: 2 }] };++ for (let i = 0; i < 5000; i++) {+ const p = new MutationJournal().track(state);+ p.seen = p.seen || {};+ p.items = p.items.filter(() => true);+ }++ expect(JSON.stringify(state)).toBe('{"seen":{},"items":[{"id":1},{"id":2}]}');+ expect(types.isProxy(state.seen)).toBe(false);+ expect(types.isProxy(state.items[0])).toBe(false);+ });++ test('hands out Map and Set untracked: they keep working and their changes stay', () => {+ const target = { ids: new Set<string>(), byId: new Map<string, number>() };+ const journal = new MutationJournal();+ const proxy = journal.track(target);++ proxy.ids.add('a');+ proxy.byId.set('a', 1);+ journal.revert();++ expect([...target.ids]).toEqual(['a']);+ expect([...target.byId]).toEqual([['a', 1]]);+ });++ test('undoOnto reconstructs the pre-attempt shape on a copy without touching the original', () => {+ const target = { count: 0, nested: { keep: 1, drop: 2 }, list: ['a'], replaced: { x: 1 } };+ const journal = new MutationJournal();+ const proxy = journal.track(target);++ proxy.count++;+ delete (proxy.nested as Partial<typeof target.nested>).drop;+ proxy.list.push('b');+ proxy.replaced = { y: 2 } as unknown as typeof target.replaced;+ (proxy.replaced as Record<string, unknown>).z = 3;++ const copy = journal.undoOnto(JSON.parse(JSON.stringify(target)));++ expect(copy).toEqual({ count: 0, nested: { keep: 1, drop: 2 }, list: ['a'], replaced: { x: 1 } });+ expect(target).toEqual({ count: 1, nested: { keep: 1 }, list: ['a', 'b'], replaced: { y: 2, z: 3 } });+ });++ test('leaves a property that was written by someone else and reports the conflict', () => {+ const target = { mine: 0, theirs: 0 };+ const journal = new MutationJournal();+ const proxy = journal.track(target);++ proxy.mine = 1;+ proxy.theirs = 1;+ target.theirs = 2;++ const conflicts: PropertyKey[] = [];+ journal.revert({ onConflict: (key) => conflicts.push(key) });++ expect(target).toEqual({ mine: 0, theirs: 2 });+ expect(conflicts).toEqual(['theirs']);+ });++ test('stops recording once closed while writes still go through', () => {+ const target = { value: 0 };+ const journal = new MutationJournal();+ const proxy = journal.track(target);++ proxy.value = 1;+ journal.close();+ proxy.value = 2;+ journal.revert();++ expect(target.value).toBe(2);+ });++ test('covers request accessors backed by private members, userData replacement and the internal bag', () => {+ const request = new CrawlingRequest<Record<string, unknown>>({+ url: 'https://example.com',+ userData: { a: 1 },+ });+ const journal = new MutationJournal();+ const proxy = journal.track(request);++ proxy.noRetry = true;+ proxy.label = 'later';+ proxy.skipNavigation = true;+ proxy.userData.a = 2;+ proxy.userData = { replaced: true };+ proxy.userData.b = 3;++ expect(request.skipNavigation).toBe(true);+ journal.revert();++ expect(request.noRetry).toBe(false);+ expect(request.label).toBeUndefined();+ expect(request.skipNavigation).toBe(false);+ expect(request.userData).toEqual({ a: 1 });+ expect(JSON.parse(JSON.stringify(request)).userData).toMatchObject({ a: 1 });+ });+});