Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
fix(playwright): track and drain in-flight rendering detections on teardown (#4100)
Yash Hirgude8abbb5d41308
diff --git a/docs/public-api/crawlee-playwright.api.md b/docs/public-api/crawlee-playwright.api.mdindex d1e3ed444..d04f45b77 100644--- a/docs/public-api/crawlee-playwright.api.md+++ b/docs/public-api/crawlee-playwright.api.md@@ -74,11 +74,14 @@ export class AdaptivePlaywrightCrawler<ContextExtension = Dictionary<never>, Extconstructor(options?: AdaptivePlaywrightCrawlerOptions<ContextExtension, ExtendedContext, Routes, StatisticStateExtension>);// (undocumented)protected buildContextPipeline(): ContextPipeline_2<CrawlingContext_2, AdaptivePlaywrightCrawlerContext>;+ drainRenderingDetections(input?: {+ timeoutMillis?: number;+ }): Promise<void>;+ get inFlightRenderingTypeDetectionCount(): number;// (undocumented)protected init(): Promise<void>;// (undocumented)protected runRequestHandler(crawlingContext: CrawlingContext_2): Promise<void>;- // (undocumented)teardown(): Promise<void>;}@@ -253,11 +256,11 @@ function injectJQuery(page: Page, options?: {// @publicexport interface IRenderingTypePredictor {- predict(request: Request_2): {+ predict(request: Request_2): Awaitable<{renderingType: RenderingType;detectionProbabilityRecommendation: number;- };- storeResult(requests: Request_2 | Request_2[], renderingType: RenderingType): void;+ }>;+ storeResult(requests: Request_2 | Request_2[], renderingType: RenderingType): Awaitable<void>;}// @publicdiff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.mdindex bef92b89c..94fb092fb 100644--- a/docs/upgrading/upgrading_v4.md+++ b/docs/upgrading/upgrading_v4.md@@ -1223,7 +1223,7 @@ If you rely on Crawlee's default configuration (one browser context per session,### Custom rendering type predictors via the `IRenderingTypePredictor` interface-The `renderingTypePredictor` option of `AdaptivePlaywrightCrawler` is now typed as the new `IRenderingTypePredictor` interface — `predict(request)` and `storeResult(requests, renderingType)`, nothing else. The built-in `RenderingTypePredictor` implements it, so passing one still works.+The `renderingTypePredictor` option of `AdaptivePlaywrightCrawler` is now typed as the new `IRenderingTypePredictor` interface — `predict(request)` and `storeResult(requests, renderingType)`, nothing else, either of which may return a promise. The built-in `RenderingTypePredictor` implements it, so passing one still works.What changed is the lifecycle: the crawler used to call `initialize()` on the predictor it was given, even though it did not create it. It now follows the same own-only-what-you-built rule as the session and browser pools — a predictor you pass in is *borrowed*, so setting it up is your job, and `initialize` is not part of the interface at all. The built-in predictor restores its persisted state in `initialize()` and will throw `Recoverable state has not yet been loaded` from `predict()` if it is never called:@@ -1244,6 +1244,10 @@ const crawler = new AdaptivePlaywrightCrawler({If you don't pass a predictor, nothing changes: the crawler builds one from `renderingTypeDetectionRatio` and, since it owns that one, initializes it for you.+**Asynchronous predictors** — `predict()` is awaited before the crawler routes the request, so prefer loading whatever it needs up front over per-request I/O. `storeResult()` is *not* awaited per detection: the crawler tracks the promise it returns and drains everything still pending in `teardown()`, so a predictor that batches its writes can rely on them landing before the crawl ends. That wait is bounded by the internal timeout (`CRAWLEE_INTERNAL_TIMEOUT`), `crawler.drainRenderingDetections()` performs it on demand, and `crawler.inFlightRenderingTypeDetectionCount` reports what is still outstanding.++`teardown()` stops new detections from starting before it drains, so ending a `keepAlive` crawl by tearing it down from outside `run()` cannot leave a detection unpersisted either.+### Remove `experimentalContainers` optionThis experimental option relied on an outdated manifest version for browser extensions, it is not possible to achieve this with the currently supported versions.diff --git a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.tsindex 10f17c677..c2ae96744 100644--- a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts+++ b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts@@ -6,6 +6,7 @@ import type {RouteSchemas,RoutesFromSchemas,} from '@crawlee/browser';+import { setTimeout as delay } from 'node:timers/promises';import { isDeepStrictEqual } from 'node:util';import type { BasicCrawlerOptions } from '@crawlee/basic';@@ -322,6 +323,16 @@ export class AdaptivePlaywrightCrawler<*/readonly #attemptWritePolicy: Partial<StorageWritePolicy>;+ /**+ * In-flight rendering type detections, plus the pending results of an asynchronous `storeResult`.+ */+ readonly #activeDetections = new Set<Promise<unknown>>();++ /**+ * Set once `teardown()` starts, so that requests still in the pool stop opening new detections.+ */+ #shutDown = false;+#teardownHooks: (() => Promise<unknown>)[] = [];constructor(@@ -482,6 +493,8 @@ export class AdaptivePlaywrightCrawler<}protected override async init(): Promise<void> {+ // A crawler can be run again after a teardown.+ this.#shutDown = false;// Only the predictor we built ourselves is ours to initialize - an injected one is borrowed, so its// lifecycle (including restoring persisted state) stays with whoever created it.await this.#renderingTypePredictor.ifOwned((predictor) => predictor.initialize());@@ -656,8 +669,9 @@ export class AdaptivePlaywrightCrawler<}protected override async runRequestHandler(crawlingContext: CrawlingContext): Promise<void> {- const renderingTypePrediction = this.#renderingTypePredictor.value.predict(crawlingContext.request);- const shouldDetectRenderingType = Math.random() < renderingTypePrediction.detectionProbabilityRecommendation;+ const renderingTypePrediction = await this.#renderingTypePredictor.value.predict(crawlingContext.request);+ const shouldDetectRenderingType =+ !this.#shutDown && Math.random() < renderingTypePrediction.detectionProbabilityRecommendation;if (!shouldDetectRenderingType) {crawlingContext.log.debug(@@ -753,40 +767,61 @@ export class AdaptivePlaywrightCrawler<await browserRun.result.commit();if (shouldDetectRenderingType) {- crawlingContext.log.debug(`Detecting rendering type for ${crawlingContext.request.url}`);- // The detection attempt's transaction is never committed - its writes exist only for the- // result comparison.- const plainHTTPRun = await this.crawlOne(- 'static',- crawlingContext,- stateTracker.getStateCopy.bind(stateTracker),- transactions,- );+ const detectionPromise = (async () => {+ crawlingContext.log.debug(`Detecting rendering type for ${crawlingContext.request.url}`);+ // The detection attempt's transaction is never committed - its writes exist only for the+ // result comparison.+ const plainHTTPRun = await this.crawlOne(+ 'static',+ crawlingContext,+ stateTracker.getStateCopy.bind(stateTracker),+ transactions,+ );- const detectionResult: RenderingType | undefined = (() => {- if (!plainHTTPRun.ok) {- return 'clientOnly';- }+ const detectionResult: RenderingType | undefined = (() => {+ if (!plainHTTPRun.ok) {+ return 'clientOnly';+ }- const comparisonResult = this.#resultComparator(plainHTTPRun.result, browserRun.result);- if (comparisonResult === true || comparisonResult === 'equal') {- return 'static';- }+ const comparisonResult = this.#resultComparator(plainHTTPRun.result, browserRun.result);+ if (comparisonResult === true || comparisonResult === 'equal') {+ return 'static';+ }- if (comparisonResult === false || comparisonResult === 'different') {- return 'clientOnly';- }+ if (comparisonResult === false || comparisonResult === 'different') {+ return 'clientOnly';+ }- return undefined;- })();+ return undefined;+ })();- crawlingContext.log.debug(- `Detected rendering type ${detectionResult} for ${crawlingContext.request.url}`,- );+ crawlingContext.log.debug(+ `Detected rendering type ${detectionResult} for ${crawlingContext.request.url}`,+ );- if (detectionResult !== undefined) {- this.#renderingTypePredictor.value.storeResult(crawlingContext.request, detectionResult);- }+ if (detectionResult !== undefined) {+ // Deliberately not awaited: a predictor that persists asynchronously gets to keep+ // batching its writes, and the drain below catches whatever is still pending.+ const stored = this.#renderingTypePredictor.value.storeResult(+ crawlingContext.request,+ detectionResult,+ );++ if (stored !== undefined) {+ // Nothing downstream awaits this, so a failed write would otherwise be silent.+ void this.#trackDetection(+ Promise.resolve(stored).catch((error) =>+ this.log.exception(+ error as Error,+ `Failed to store the rendering type detection result for ${crawlingContext.request.url}`,+ ),+ ),+ );+ }+ }+ })();++ await this.#trackDetection(detectionPromise);}} finally {// A still-open transaction here belongs to a discarded attempt - roll it back, then release.@@ -839,7 +874,76 @@ export class AdaptivePlaywrightCrawler<});}+ #trackDetection<T>(promise: Promise<T>): Promise<T> {+ this.#activeDetections.add(promise);+ // Not `finally()`: the promise it derives would reject on its own and go unhandled. A rejection here+ // belongs to whoever awaits the original, or to `allSettled` in the drain.+ void promise.catch(() => {}).then(() => this.#activeDetections.delete(promise));+ return promise;+ }++ /**+ * Number of rendering type detections that have not settled yet, including results the predictor is+ * still persisting.+ */+ get inFlightRenderingTypeDetectionCount(): number {+ return this.#activeDetections.size;+ }++ /**+ * Waits for in-flight rendering type detections to settle, bounded by `timeoutMillis` (defaults to the+ * internal timeout).+ */+ async drainRenderingDetections({ timeoutMillis }: { timeoutMillis?: number } = {}): Promise<void> {+ if (this.#activeDetections.size === 0) {+ return;+ }++ const drained = (async () => {+ while (this.#activeDetections.size > 0) {+ await Promise.allSettled(Array.from(this.#activeDetections));+ }+ })();++ const millis = timeoutMillis ?? this.internalTimeoutMillis;++ // A caller opting out of the bound would otherwise get an immediate spurious timeout - `setTimeout`+ // clamps a non-finite delay to 1ms.+ if (!Number.isFinite(millis)) {+ await drained;+ return;+ }++ const abortTimer = new AbortController();++ try {+ const outcome = await Promise.race([+ drained.then(() => 'drained' as const),+ delay(millis, 'timedOut' as const, { signal: abortTimer.signal }).catch(() => 'aborted' as const),+ ]);++ if (outcome === 'timedOut') {+ this.log.warning(+ `Timed out after ${millis / 1e3} seconds waiting for ${this.#activeDetections.size} rendering type detection(s) to settle - their results may be lost.`,+ );+ }+ } finally {+ abortTimer.abort();+ }+ }++ /**+ * Stops the crawler immediately, but not before rendering type detections already under way (and results+ * the predictor is still persisting) have settled - see+ * {@apilink AdaptivePlaywrightCrawler.drainRenderingDetections|`drainRenderingDetections()`}. Requests+ * that are still running are not waited for, unlike {@apilink BasicCrawler.stop|`stop()`}.+ */override async teardown() {+ // Called from outside `run()` - under `keepAlive`, say - the pool keeps dispatching until+ // `super.teardown()` aborts it, and a request starting during the drain would open a detection the+ // drain has already passed. Closing that first makes the drain a fence.+ this.#shutDown = true;+ await this.drainRenderingDetections();await super.teardown();// Mirrors the owned-only `initialize()` in `init()` - without this, the predictor we built keeps its// PERSIST_STATE listener registered after the crawl and never gets a final write.diff --git a/packages/playwright-crawler/src/internals/utils/rendering-type-prediction.ts b/packages/playwright-crawler/src/internals/utils/rendering-type-prediction.tsindex 3f6d14664..55f1972d5 100644--- a/packages/playwright-crawler/src/internals/utils/rendering-type-prediction.ts+++ b/packages/playwright-crawler/src/internals/utils/rendering-type-prediction.ts@@ -1,5 +1,6 @@import type { RecoverableStatePersistenceOptions, Request } from '@crawlee/core';import { RecoverableState } from '@crawlee/core';+import type { Awaitable } from '@crawlee/types';import LogisticRegression from 'ml-logistic-regression';import { Matrix } from 'ml-matrix';import stringComparison from 'string-comparison';@@ -54,14 +55,25 @@ export interface RenderingTypePredictorOptions {* @experimental*/export interface IRenderingTypePredictor {- /** Predict the rendering type for a request, and how likely the crawler should be to verify it. */- predict(request: Request): {+ /**+ * Predict the rendering type for a request, and how likely the crawler should be to verify it.+ *+ * Called once per request, before navigation - the crawler awaits it, so prefer loading whatever the+ * prediction needs up front over per-request I/O.+ */+ predict(request: Request): Awaitable<{renderingType: RenderingType;detectionProbabilityRecommendation: number;- };+ }>;- /** Report a detected rendering type, so that future predictions can take it into account. */- storeResult(requests: Request | Request[], renderingType: RenderingType): void;+ /**+ * Report a detected rendering type, so that future predictions can take it into account.+ *+ * The crawler does not await this per detection - it collects the returned promises and drains them in+ * `teardown()`, so an implementation that persists results asynchronously can keep batching its writes+ * and still be sure they land before the crawl ends.+ */+ storeResult(requests: Request | Request[], renderingType: RenderingType): Awaitable<void>;}const renderingType = z.enum(['clientOnly', 'static'] as const satisfies readonly RenderingType[]);diff --git a/test/core/crawlers/adaptive_playwright_crawler.test.ts b/test/core/crawlers/adaptive_playwright_crawler.test.tsindex d5cd99677..20a8c31a9 100644--- a/test/core/crawlers/adaptive_playwright_crawler.test.ts+++ b/test/core/crawlers/adaptive_playwright_crawler.test.ts@@ -3,6 +3,7 @@ import type { AddressInfo } from 'node:net';import {BaseCrawleeLogger,+ Configuration,type CrawleeLogger,type CrawleeLoggerOptions,Dataset,@@ -1023,4 +1024,296 @@ describe('AdaptivePlaywrightCrawler', () => {expect(lastDynamicRequestUserAgent).toBe(distinctiveUserAgent);});++ describe('in-flight rendering type detections', () => {+ test('are counted while running and settled once the crawl ends', async () => {+ const renderingTypePredictor = makeRiggedRenderingTypePredictor({+ detectionProbabilityRecommendation: 1,+ renderingType: 'clientOnly',+ });++ // A detection re-runs the user handler over plain HTTP, so the handler can observe whether+ // it is itself running inside a detection.+ const observedCounts: number[] = [];++ const crawler = await makeOneshotCrawler(+ {+ requestHandler: async () => {+ observedCounts.push(crawler.inFlightRenderingTypeDetectionCount);+ },+ renderingTypePredictor,+ },+ [`http://${HOSTNAME}:${port}/static`],+ );++ await crawler.run();++ expect(observedCounts).toEqual([0, 1]);+ expect(crawler.inFlightRenderingTypeDetectionCount).toBe(0);+ });++ // A crawler whose plain-HTTP detection attempt parks until released, so that a detection can be+ // observed mid-flight.+ const makeCrawlerWithBlockedDetection = async () => {+ const renderingTypePredictor = makeRiggedRenderingTypePredictor({+ detectionProbabilityRecommendation: 1,+ renderingType: 'clientOnly',+ });++ let releaseDetection!: () => void;+ const detectionReleased = new Promise<void>((resolve) => {+ releaseDetection = resolve;+ });+ let announceDetection!: () => void;+ const detectionStarted = new Promise<void>((resolve) => {+ announceDetection = resolve;+ });++ let handlerCalls = 0;+ const crawler = await makeOneshotCrawler(+ {+ requestHandler: async () => {+ handlerCalls += 1;+ // The second call is the plain-HTTP detection attempt.+ if (handlerCalls === 2) {+ announceDetection();+ await detectionReleased;+ }+ },+ renderingTypePredictor,+ },+ [`http://${HOSTNAME}:${port}/static`],+ );++ return { crawler, renderingTypePredictor, detectionStarted, releaseDetection };+ };++ test('hold up drainRenderingDetections until their result is stored', async () => {+ const { crawler, renderingTypePredictor, detectionStarted, releaseDetection } =+ await makeCrawlerWithBlockedDetection();++ const runPromise = crawler.run();+ await detectionStarted;++ expect(crawler.inFlightRenderingTypeDetectionCount).toBe(1);++ let drained = false;+ const drainPromise = crawler.drainRenderingDetections().then(() => {+ drained = true;+ });++ await sleep(100);+ expect(drained).toBe(false);+ expect(renderingTypePredictor.storeResult).not.toHaveBeenCalled();++ releaseDetection();+ await drainPromise;++ expect(drained).toBe(true);+ expect(crawler.inFlightRenderingTypeDetectionCount).toBe(0);+ expect(renderingTypePredictor.storeResult).toHaveBeenCalledOnce();++ await runPromise;+ });++ test('hold up teardown until their result is stored', async () => {+ const { crawler, renderingTypePredictor, detectionStarted, releaseDetection } =+ await makeCrawlerWithBlockedDetection();++ const runPromise = crawler.run();+ await detectionStarted;++ let tornDown = false;+ const teardownPromise = crawler.teardown().then(() => {+ tornDown = true;+ });++ await sleep(100);+ expect(tornDown).toBe(false);++ releaseDetection();+ await teardownPromise;++ expect(renderingTypePredictor.storeResult).toHaveBeenCalledOnce();++ await runPromise;+ });++ test('an asynchronous predict decides the rendering path', async () => {+ const crawler = await makeOneshotCrawler(+ {+ requestHandler: async () => {},+ renderingTypePredictor: {+ predict: async () => {+ await sleep(10);+ return { detectionProbabilityRecommendation: 0, renderingType: 'static' as const };+ },+ storeResult: () => {},+ },+ },+ [`http://${HOSTNAME}:${port}/static`],+ );++ await crawler.run();++ // An unawaited prediction would read `undefined` off the promise and fall through to the browser.+ expect(crawler.statistics.state.httpOnlyRequestHandlerRuns).toBe(1);+ expect(crawler.statistics.state.browserRequestHandlerRuns).toBe(0);+ });++ test('a pending asynchronous storeResult holds up the crawl', async () => {+ let resolveStore!: () => void;+ const storeFinished = new Promise<void>((resolve) => {+ resolveStore = resolve;+ });+ const storeResult = vi.fn(() => storeFinished);++ const crawler = await makeOneshotCrawler(+ {+ requestHandler: async () => {},+ renderingTypePredictor: {+ predict: () => ({+ detectionProbabilityRecommendation: 1,+ renderingType: 'clientOnly' as const,+ }),+ storeResult,+ },+ },+ [`http://${HOSTNAME}:${port}/static`],+ );++ let finished = false;+ const runPromise = crawler.run().then(() => {+ finished = true;+ });++ await vi.waitFor(() => expect(storeResult).toHaveBeenCalledOnce());+ await sleep(100);++ // The detection itself is long done - what the crawl is waiting on is the predictor's write.+ expect(finished).toBe(false);+ expect(crawler.inFlightRenderingTypeDetectionCount).toBe(1);++ resolveStore();+ await runPromise;++ expect(finished).toBe(true);+ expect(crawler.inFlightRenderingTypeDetectionCount).toBe(0);+ });++ test('a storeResult that never settles is abandoned after the internal timeout', async () => {+ const messages: string[] = [];++ serviceLocator.setConfiguration(new Configuration({ internalTimeoutMillis: 250 }));++ const crawler = await makeOneshotCrawler(+ {+ requestHandler: async () => {},+ renderingTypePredictor: {+ predict: () => ({+ detectionProbabilityRecommendation: 1,+ renderingType: 'clientOnly' as const,+ }),+ storeResult: () => new Promise<void>(() => {}),+ },+ logger: new RecordingLogger(messages),+ },+ [`http://${HOSTNAME}:${port}/static`],+ );++ await crawler.run();++ // The bound comes from `internalTimeoutMillis`, not an option of its own.+ expect(messages).toContainEqual(+ expect.stringContaining('Timed out after 0.25 seconds waiting for 1 rendering type detection(s)'),+ );+ });++ test('a rejected asynchronous storeResult is reported and does not fail the request', async () => {+ const messages: string[] = [];++ const crawler = await makeOneshotCrawler(+ {+ requestHandler: async () => {},+ renderingTypePredictor: {+ predict: () => ({+ detectionProbabilityRecommendation: 1,+ renderingType: 'clientOnly' as const,+ }),+ storeResult: async () => {+ throw new Error('the write failed');+ },+ },+ logger: new RecordingLogger(messages),+ },+ [`http://${HOSTNAME}:${port}/static`],+ );++ const stats = await crawler.run();++ expect(stats.requestsFailed).toBe(0);+ expect(messages).toContainEqual(+ expect.stringContaining('Failed to store the rendering type detection result'),+ );+ });++ // Under `keepAlive` the crawl is ended by an external `teardown()`, with the pool still dispatching.+ test('teardown stops requests that start during the drain from opening new detections', async () => {+ const firstUrl = `http://${HOSTNAME}:${port}/static?q=first`;+ const secondUrl = `http://${HOSTNAME}:${port}/static?q=second`;++ let releaseDetection!: () => void;+ const detectionReleased = new Promise<void>((resolve) => {+ releaseDetection = resolve;+ });+ let announceDetection!: () => void;+ const detectionStarted = new Promise<void>((resolve) => {+ announceDetection = resolve;+ });++ // Keeps the drain pending for as long as the test needs, so the pool is guaranteed to reach the+ // second request while `teardown()` is still inside it.+ let releaseStore!: () => void;+ const storeFinished = new Promise<void>((resolve) => {+ releaseStore = resolve;+ });+ const storeResult = vi.fn(() => storeFinished);++ const handled: string[] = [];+ const crawler = new AdaptivePlaywrightCrawler({+ keepAlive: true,+ maxConcurrency: 1,+ maxRequestRetries: 0,+ requestList: await RequestList.open({ sources: [firstUrl, secondUrl] }),+ renderingTypePredictor: {+ predict: () => ({ detectionProbabilityRecommendation: 1, renderingType: 'clientOnly' }),+ storeResult,+ },+ requestHandler: async ({ request }) => {+ handled.push(request.url);++ // The second run of the first request is its detection attempt.+ if (request.url === firstUrl && handled.filter((url) => url === firstUrl).length === 2) {+ announceDetection();+ await detectionReleased;+ }+ },+ });++ const runPromise = crawler.run();+ await detectionStarted;++ const teardownPromise = crawler.teardown();+ releaseDetection();++ await vi.waitFor(() => expect(handled).toContain(secondUrl));+ expect(storeResult).toHaveBeenCalledOnce();++ releaseStore();+ await teardownPromise;+ await runPromise;++ expect(storeResult).toHaveBeenCalledOnce();+ });+ });});