Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

feat: Accept custom extensions in the IStatistics interface (#3982)

Jan Buchare00e8ce84fba
diff --git a/docs/guides/custom_statistics.mdx b/docs/guides/custom_statistics.mdx
new file mode 100644
index 000000000..49227ec85
--- /dev/null
+++ b/docs/guides/custom_statistics.mdx
@@ -0,0 +1,52 @@
+---
+id: custom-statistics
+title: Custom statistics fields
+description: Track your own counters alongside the built-in crawler statistics
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+import CodeBlock from '@theme/CodeBlock';
+
+import BasicSource from '!!raw-loader!./custom_statistics_basic.ts';
+import ValidatedSource from '!!raw-loader!./custom_statistics_validated.ts';
+import AdaptiveSource from '!!raw-loader!./custom_statistics_adaptive.ts';
+
+Every crawler collects run statistics — requests finished, failures, retries, durations — into a <ApiLink to="core/class/Statistics">`Statistics`</ApiLink> instance exposed as `crawler.stats`. You can track your own counters in the same place, and get the persistence, logging and lifecycle handling that the built-in fields already have.
+
+Keeping a counter in a plain variable works right up until the run is interrupted. Statistics state is written to the key-value store and restored on migration or resurrect, so a counter that lives in `stats.state` survives a run that a plain variable would not.
+
+## Declaring custom fields
+
+Pass <ApiLink to="core/interface/StatisticsOptions#stateExtension">`stateExtension`</ApiLink> when constructing a <ApiLink to="core/class/Statistics">`Statistics`</ApiLink> instance, and inject it into the crawler via the `statistics` option. The fields become part of `state`, typed exactly as you declared them:
+
+<CodeBlock className="language-typescript">{BasicSource}</CodeBlock>
+
+The field types flow through to `crawler.stats.state`, so they are available anywhere you hold the crawler, not just where you hold the statistics instance. Reading an undeclared field is a compile-time error.
+
+<ApiLink to="core/interface/StatisticStateExtensionOptions#defaultState">`defaultState`</ApiLink> is also what `reset()` restores, and it determines which keys are read back from a persisted record. A field you add later simply keeps its default when an older record is restored, so growing the state between runs does not need a migration.
+
+## Validating the fields on the way back
+
+Declaring `defaultState` alone means a restored record is taken at its word: whatever `productsFound` holds in the key-value store becomes the state, and an increment on a string that says it is a number poisons every later one. Pass a <ApiLink to="core/interface/StatisticStateExtensionOptions#deserialize">`deserialize`</ApiLink> conversion to check the record before trusting it — either a plain function or, as here, a [Standard Schema](https://standardschema.dev):
+
+<CodeBlock className="language-typescript">{ValidatedSource}</CodeBlock>
+
+A schema that gives every field a default doubles as the declaration of the fields, so `defaultState` can be omitted — one place to add a field, and the defaults cannot drift from the conversion that has to accept them back. If the record fails to validate, the custom fields start from their defaults with a warning; the built-in statistics are unaffected.
+
+The fields have to be JSON-serializable, because the state round-trips through the key-value store. A `Date` or a `Set` comes back from that trip as a string or a plain object, so either keep to numbers, strings, booleans, `null`, and plain arrays and objects of those, or pair `deserialize` with a <ApiLink to="core/interface/StatisticStateExtensionOptions#serialize">`serialize`</ApiLink> that converts them, as `lastSeenAt` does above.
+
+:::note Injected instances are not reset
+
+A crawler resets the statistics it built itself at the start of every `run()`, but never one you passed in — a supplied instance keeps whatever state it was handed. If you reuse a crawler for several runs and want the counters to start from zero each time, call `statistics.reset()` yourself.
+
+:::
+
+## Crawlers that track fields of their own
+
+<ApiLink to="playwright-crawler/class/AdaptivePlaywrightCrawler">`AdaptivePlaywrightCrawler`</ApiLink> tracks a few extra fields itself (how many requests were handled with and without a browser, and how many rendering type mispredictions occurred). It builds a suitable `Statistics` instance by default, so you only need to do anything if you want to add fields of your own — in which case your declaration has to include the crawler's fields as well:
+
+<CodeBlock className="language-typescript">{AdaptiveSource}</CodeBlock>
+
+## Custom statistics implementations
+
+If you need something more than extra fields — reporting to an external metrics backend, for example — the `statistics` option accepts any object implementing <ApiLink to="core/interface/IStatistics">`IStatistics`</ApiLink>, not just the built-in class. The crawler drives it purely through that interface.
diff --git a/docs/guides/custom_statistics_adaptive.ts b/docs/guides/custom_statistics_adaptive.ts
new file mode 100644
index 000000000..69b0c29e7
--- /dev/null
+++ b/docs/guides/custom_statistics_adaptive.ts
@@ -0,0 +1,22 @@
+import { AdaptivePlaywrightCrawler, adaptivePlaywrightCrawlerStatisticState, Statistics } from 'crawlee';
+import { z } from 'zod';
+
+const statistics = new Statistics({
+ stateExtension: {
+ // The adaptive crawler tracks fields of its own, so an injected instance has to carry them too.
+ deserialize: adaptivePlaywrightCrawlerStatisticState.deserialize.extend({
+ productsFound: z.number().default(0),
+ }),
+ },
+});
+
+const crawler = new AdaptivePlaywrightCrawler({
+ statistics,
+ async requestHandler({ querySelectorAll }) {
+ statistics.state.productsFound += (await querySelectorAll('.product')).length;
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
+
+console.log(`Handled ${crawler.stats.state.httpOnlyRequestHandlerRuns} requests without a browser`);
diff --git a/docs/guides/custom_statistics_basic.ts b/docs/guides/custom_statistics_basic.ts
new file mode 100644
index 000000000..643b9a32f
--- /dev/null
+++ b/docs/guides/custom_statistics_basic.ts
@@ -0,0 +1,20 @@
+import { CheerioCrawler, Statistics } from 'crawlee';
+
+// Declare the extra fields and their initial values. Their types flow into `statistics.state`.
+const statistics = new Statistics({
+ stateExtension: { defaultState: { productsFound: 0 } },
+});
+
+const crawler = new CheerioCrawler({
+ statistics,
+ async requestHandler({ $, enqueueLinks }) {
+ statistics.state.productsFound += $('.product').length;
+
+ await enqueueLinks();
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
+
+// The custom fields are typed on `crawler.stats` too.
+console.log(`Found ${crawler.stats.state.productsFound} products`);
diff --git a/docs/guides/custom_statistics_validated.ts b/docs/guides/custom_statistics_validated.ts
new file mode 100644
index 000000000..c05a77eff
--- /dev/null
+++ b/docs/guides/custom_statistics_validated.ts
@@ -0,0 +1,27 @@
+import { CheerioCrawler, Statistics } from 'crawlee';
+import { z } from 'zod';
+
+const statistics = new Statistics({
+ stateExtension: {
+ // A default per field means the schema is the only place the fields are declared.
+ deserialize: z.object({
+ productsFound: z.number().default(0),
+ lastSeenAt: z.coerce.date().default(() => new Date()),
+ }),
+ // `Date` is not JSON - this is how it gets into the record for `z.coerce.date()` to read back.
+ serialize: ({ productsFound, lastSeenAt }) => ({
+ productsFound,
+ lastSeenAt: lastSeenAt.toISOString(),
+ }),
+ },
+});
+
+const crawler = new CheerioCrawler({
+ statistics,
+ async requestHandler({ $ }) {
+ statistics.state.productsFound += $('.product').length;
+ statistics.state.lastSeenAt = new Date();
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
diff --git a/docs/public-api/crawlee-basic.api.md b/docs/public-api/crawlee-basic.api.md
index c709b6431..3e65c021d 100644
--- a/docs/public-api/crawlee-basic.api.md
+++ b/docs/public-api/crawlee-basic.api.md
@@ -51,8 +51,8 @@ import type { TypedRequestsLike } from '@crawlee/core';
import { z } from 'zod';
// @public (undocumented)
-export class BasicCrawler<Context extends CrawlingContext = CrawlingContext, ContextExtension = Dictionary<never>, ExtendedContext extends Context = Context & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>> {
- constructor(options?: BasicCrawlerOptions<Context, ContextExtension, ExtendedContext, Routes> & RequireContextPipeline<CrawlingContext, Context>);
+export class BasicCrawler<Context extends CrawlingContext = CrawlingContext, ContextExtension = Dictionary<never>, ExtendedContext extends Context = Context & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>, StatisticStateExtension extends object = {}> {
+ constructor(options?: BasicCrawlerOptions<Context, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> & RequireContextPipeline<CrawlingContext, Context>);
// (undocumented)
protected readonly additionalHttpErrorStatusCodes: Set<number>;
addRequests(requests: ReadonlyDeep<TypedRequestsLike<Routes>>, options?: CrawlerAddRequestsOptions): Promise<CrawlerAddRequestsResult>;
@@ -219,7 +219,7 @@ export class BasicCrawler<Context extends CrawlingContext = CrawlingContext, Con
protected runRequestHandler(crawlingContext: ExtendedContext): Promise<void>;
get sessionPool(): ISessionPool;
setStatusMessage(message: string, options?: SetStatusMessageOptions): void;
- get stats(): IStatistics;
+ get stats(): IStatistics<StatisticStateExtension>;
stop(reason?: string): void;
teardown(): Promise<void>;
protected throwOnBlockedRequest(statusCode: number): void;
@@ -228,7 +228,7 @@ export class BasicCrawler<Context extends CrawlingContext = CrawlingContext, Con
}
// @public (undocumented)
-export interface BasicCrawlerOptions<Context extends CrawlingContext = CrawlingContext, ContextExtension = Dictionary<never>, ExtendedContext extends Context = Context & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>> {
+export interface BasicCrawlerOptions<Context extends CrawlingContext = CrawlingContext, ContextExtension = Dictionary<never>, ExtendedContext extends Context = Context & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>, StatisticStateExtension extends object = {}> {
additionalHttpErrorStatusCodes?: number[];
blockedStatusCodes?: number[];
concurrencySystem?: IConcurrencySystem;
@@ -264,7 +264,7 @@ export interface BasicCrawlerOptions<Context extends CrawlingContext = CrawlingC
retryOnBlocked?: boolean;
sameDomainDelaySecs?: number;
sessionPool?: ISessionPool;
- statistics?: IStatistics;
+ statistics?: IStatistics<StatisticStateExtension>;
statusMessageCallback?: StatusMessageCallback;
statusMessageLoggingInterval?: number;
storageBackend?: StorageBackend;
diff --git a/docs/public-api/crawlee-browser.api.md b/docs/public-api/crawlee-browser.api.md
index c9634ef49..a4b3e6cf0 100644
--- a/docs/public-api/crawlee-browser.api.md
+++ b/docs/public-api/crawlee-browser.api.md
@@ -43,8 +43,8 @@ interface BaseResponse {
}
// @public
-export abstract class BrowserCrawler<Page extends CommonPage = CommonPage, Response extends BaseResponse = BaseResponse, InternalBrowserPoolOptions extends BrowserPoolOptions = BrowserPoolOptions, LaunchOptions extends Dictionary | undefined = Dictionary, Context extends BrowserCrawlingContext<Page, Response> = BrowserCrawlingContext<Page, Response>, ContextExtension = Dictionary<never>, ExtendedContext extends Context = Context & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>, GoToOptions extends Dictionary = Dictionary> extends BasicCrawler<Context, ContextExtension, ExtendedContext, Routes> {
- protected constructor(options: BrowserCrawlerOptions<Page, Response, Context, ContextExtension, ExtendedContext> & {
+export abstract class BrowserCrawler<Page extends CommonPage = CommonPage, Response extends BaseResponse = BaseResponse, InternalBrowserPoolOptions extends BrowserPoolOptions = BrowserPoolOptions, LaunchOptions extends Dictionary | undefined = Dictionary, Context extends BrowserCrawlingContext<Page, Response> = BrowserCrawlingContext<Page, Response>, ContextExtension = Dictionary<never>, ExtendedContext extends Context = Context & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>, StatisticStateExtension extends object = {}, GoToOptions extends Dictionary = Dictionary> extends BasicCrawler<Context, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
+ protected constructor(options: BrowserCrawlerOptions<Page, Response, Context, ContextExtension, ExtendedContext, BrowserPoolOptions, Routes, StatisticStateExtension> & {
contextPipelineBuilder: () => ContextPipeline<CrawlingContext, Context>;
});
get browserPool(): IBrowserPool<Page>;
@@ -176,7 +176,7 @@ export abstract class BrowserCrawler<Page extends CommonPage = CommonPage, Respo
}
// @public (undocumented)
-export interface BrowserCrawlerOptions<Page extends CommonPage = CommonPage, Response extends BaseResponse = BaseResponse, Context extends BrowserCrawlingContext<Page, Response> = BrowserCrawlingContext<Page, Response>, ContextExtension = Dictionary<never>, ExtendedContext extends Context = Context & ContextExtension, InternalBrowserPoolOptions extends BrowserPoolOptions = BrowserPoolOptions, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>, __BrowserPlugins extends BrowserPlugin[] = InferBrowserPluginArray<InternalBrowserPoolOptions['browserPlugins']>, __BrowserControllerReturn extends BrowserController = ReturnType<__BrowserPlugins[number]['createController']>, __LaunchContextReturn extends LaunchContext = ReturnType<__BrowserPlugins[number]['createLaunchContext']>> extends Omit<BasicCrawlerOptions<Context, ContextExtension, ExtendedContext>, 'requestHandler' | 'failedRequestHandler' | 'errorHandler'> {
+export interface BrowserCrawlerOptions<Page extends CommonPage = CommonPage, Response extends BaseResponse = BaseResponse, Context extends BrowserCrawlingContext<Page, Response> = BrowserCrawlingContext<Page, Response>, ContextExtension = Dictionary<never>, ExtendedContext extends Context = Context & ContextExtension, InternalBrowserPoolOptions extends BrowserPoolOptions = BrowserPoolOptions, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>, StatisticStateExtension extends object = {}, __BrowserPlugins extends BrowserPlugin[] = InferBrowserPluginArray<InternalBrowserPoolOptions['browserPlugins']>, __BrowserControllerReturn extends BrowserController = ReturnType<__BrowserPlugins[number]['createController']>, __LaunchContextReturn extends LaunchContext = ReturnType<__BrowserPlugins[number]['createLaunchContext']>> extends Omit<BasicCrawlerOptions<Context, ContextExtension, ExtendedContext, Routes, StatisticStateExtension>, 'requestHandler' | 'failedRequestHandler' | 'errorHandler'> {
browserPool?: IBrowserPool<Page>;
browserPoolOptions?: Partial<BrowserPoolOptions> & Partial<BrowserPoolHooks<__BrowserControllerReturn, __LaunchContextReturn>>;
errorHandler?: ErrorHandler<CrawlingContext, ExtendedContext>;
diff --git a/docs/public-api/crawlee-cheerio.api.md b/docs/public-api/crawlee-cheerio.api.md
index d2740d96c..6c1168a23 100644
--- a/docs/public-api/crawlee-cheerio.api.md
+++ b/docs/public-api/crawlee-cheerio.api.md
@@ -26,8 +26,8 @@ import type { RouteSchemas } from '@crawlee/http';
import type { RoutesFromSchemas } from '@crawlee/http';
// @public
-export class CheerioCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends CheerioCrawlingContext = CheerioCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<CheerioCrawlingContext['request']>>> extends HttpCrawler<CheerioCrawlingContext, ContextExtension, ExtendedContext, Routes> {
- constructor(options?: CheerioCrawlerOptions<ContextExtension, ExtendedContext, any, any, Routes>);
+export class CheerioCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends CheerioCrawlingContext = CheerioCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<CheerioCrawlingContext['request']>>, StatisticStateExtension extends object = {}> extends HttpCrawler<CheerioCrawlingContext, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
+ constructor(options?: CheerioCrawlerOptions<ContextExtension, ExtendedContext, any, any, Routes, StatisticStateExtension>);
// (undocumented)
protected buildContextPipeline(): ContextPipeline<CrawlingContext<Dictionary>, InternalHttpCrawlingContext<any, any> & {
readonly body: string;
@@ -43,7 +43,7 @@ export class CheerioCrawler<ContextExtension = Dictionary<never>, ExtendedContex
// @public (undocumented)
export interface CheerioCrawlerOptions<ContextExtension = Dictionary<never>, ExtendedContext extends CheerioCrawlingContext = CheerioCrawlingContext & ContextExtension, UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
-Routes extends Record<keyof Routes, Dictionary> = Record<string, UserData>> extends HttpCrawlerOptions<CheerioCrawlingContext<UserData, JSONData>, ContextExtension, ExtendedContext, Routes> {
+Routes extends Record<keyof Routes, Dictionary> = Record<string, UserData>, StatisticStateExtension extends object = {}> extends HttpCrawlerOptions<CheerioCrawlingContext<UserData, JSONData>, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
}
// @public (undocumented)
diff --git a/docs/public-api/crawlee-core.api.md b/docs/public-api/crawlee-core.api.md
index 5d92452ed..8d8bdb9d2 100644
--- a/docs/public-api/crawlee-core.api.md
+++ b/docs/public-api/crawlee-core.api.md
@@ -809,7 +809,7 @@ export interface IRequestManager extends IRequestLoader {
}
// @public
-export interface IStatistics {
+export interface IStatistics<StateExtension extends object = {}> {
calculate(): CalculatedStatistics;
discardJob(id: number | string): void;
readonly errorTracker: ErrorTracker;
@@ -821,7 +821,7 @@ export interface IStatistics {
readonly requestRetryHistogram: number[];
startCapturing(): Promise<void>;
startJob(id: number | string): void;
- readonly state: StatisticState;
+ readonly state: StatisticState & StateExtension;
stopCapturing(): Promise<void>;
}
@@ -1939,30 +1939,25 @@ export interface StatisticPersistedState extends Omit<StatisticState, 'statsPers
}
// @public
-export class Statistics implements IStatistics {
- constructor(options?: StatisticsOptions);
+export class Statistics<StateExtension extends object = {}, PersistedStateExtension extends object = StateExtension> implements IStatistics<StateExtension> {
+ constructor(options?: StatisticsOptions<StateExtension, PersistedStateExtension>);
calculate(): CalculatedStatistics;
- protected defaultState(): StatisticState;
- protected deserializeState(persistedState: StatisticPersistedState): StatisticState;
readonly errorTracker: ErrorTracker;
readonly errorTrackerRetry: ErrorTracker;
readonly id: string;
persistState(): Promise<void>;
- // (undocumented)
- protected readonly persistStateKey: string;
registerStatusCode(code: number): void;
get requestRetryHistogram(): number[];
reset(): void;
resetStore(): Promise<void>;
- protected serializeState(state: StatisticState): StatisticPersistedState;
startCapturing(): Promise<void>;
- get state(): StatisticState;
+ get state(): StatisticState & StateExtension;
stopCapturing(): Promise<void>;
- toJSON(): StatisticPersistedState;
+ toJSON(): StatisticPersistedState & PersistedStateExtension;
}
// @public
-export interface StatisticsOptions {
+export interface StatisticsOptions<StateExtension extends object = {}, PersistedStateExtension extends object = StateExtension> {
id?: string;
keyValueStore?: KeyValueStore;
log?: CrawleeLogger;
@@ -1970,6 +1965,7 @@ export interface StatisticsOptions {
logMessage?: string;
persistenceOptions?: PersistenceOptions;
saveErrorSnapshots?: boolean;
+ stateExtension?: StatisticStateExtensionOptions<StateExtension, PersistedStateExtension>;
}
// @public
@@ -2010,6 +2006,13 @@ export interface StatisticState {
statsPersistedAt: Date | string | null;
}
+// @public
+export interface StatisticStateExtensionOptions<StateExtension extends object, PersistedStateExtension extends object = StateExtension> {
+ defaultState?: StateExtension | (() => StateExtension);
+ deserialize?: SyncStateConversion<unknown, StateExtension>;
+ serialize?: SyncStateConversion<StateExtension, PersistedStateExtension>;
+}
+
export { StorageBackend }
// @public
@@ -2137,6 +2140,9 @@ export interface SupportsDomainThrottling {
// @public
export function supportsDomainThrottling(manager: unknown): manager is SupportsDomainThrottling;
+// @public
+export type SyncStateConversion<TFrom, TTo> = ((value: TFrom) => TTo) | StandardSchemaV1<TFrom, TTo>;
+
// @public
export interface SystemInfo {
// (undocumented)
diff --git a/docs/public-api/crawlee-http.api.md b/docs/public-api/crawlee-http.api.md
index 336e6767a..a192462d5 100644
--- a/docs/public-api/crawlee-http.api.md
+++ b/docs/public-api/crawlee-http.api.md
@@ -100,8 +100,8 @@ export type FileDownloadRequestHandler<UserData extends Dictionary = any> = Requ
export const HTTP_OPTIMIZED_CONCURRENCY_SYSTEM_OPTIONS: ConcurrencySystemOptions;
// @public
-export class HttpCrawler<Context extends InternalHttpCrawlingContext<any, any> = InternalHttpCrawlingContext, ContextExtension = Dictionary<never>, ExtendedContext extends Context = Context & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>> extends BasicCrawler<Context, ContextExtension, ExtendedContext, Routes> {
- constructor(options?: HttpCrawlerOptions<Context, ContextExtension, ExtendedContext> & RequireContextPipeline<InternalHttpCrawlingContext, Context>);
+export class HttpCrawler<Context extends InternalHttpCrawlingContext<any, any> = InternalHttpCrawlingContext, ContextExtension = Dictionary<never>, ExtendedContext extends Context = Context & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>, StatisticStateExtension extends object = {}> extends BasicCrawler<Context, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
+ constructor(options?: HttpCrawlerOptions<Context, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> & RequireContextPipeline<InternalHttpCrawlingContext, Context>);
// (undocumented)
protected buildContextPipeline(): ContextPipeline<CrawlingContext, InternalHttpCrawlingContext>;
protected createDefaultConcurrencySystem(options: ConcurrencySystemOptions): ConcurrencySystem;
@@ -218,7 +218,7 @@ export class HttpCrawler<Context extends InternalHttpCrawlingContext<any, any> =
}
// @public (undocumented)
-export interface HttpCrawlerOptions<Context extends InternalHttpCrawlingContext = InternalHttpCrawlingContext, ContextExtension = Dictionary<never>, ExtendedContext extends Context = Context & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>> extends BasicCrawlerOptions<Context, ContextExtension, ExtendedContext, Routes> {
+export interface HttpCrawlerOptions<Context extends InternalHttpCrawlingContext = InternalHttpCrawlingContext, ContextExtension = Dictionary<never>, ExtendedContext extends Context = Context & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>, StatisticStateExtension extends object = {}> extends BasicCrawlerOptions<Context, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
additionalMimeTypes?: string[];
forceResponseEncoding?: string;
ignoreTlsErrors?: boolean;
diff --git a/docs/public-api/crawlee-jsdom.api.md b/docs/public-api/crawlee-jsdom.api.md
index 71a0e4cde..82fbb21b7 100644
--- a/docs/public-api/crawlee-jsdom.api.md
+++ b/docs/public-api/crawlee-jsdom.api.md
@@ -40,8 +40,8 @@ export function createJSDOMRouter<Context extends JSDOMCrawlingContext = JSDOMCr
export function createJSDOMRouter<Context extends JSDOMCrawlingContext = JSDOMCrawlingContext, const Schemas extends RouteSchemas = RouteSchemas>(schemas: Schemas): RouterHandler<Context, RoutesFromSchemas<Schemas>>;
// @public (undocumented)
-export class JSDOMCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends JSDOMCrawlingContext = JSDOMCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<JSDOMCrawlingContext['request']>>> extends HttpCrawler<JSDOMCrawlingContext, ContextExtension, ExtendedContext, Routes> {
- constructor(options?: JSDOMCrawlerOptions<ContextExtension, ExtendedContext, any, any, Routes>);
+export class JSDOMCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends JSDOMCrawlingContext = JSDOMCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<JSDOMCrawlingContext['request']>>, StatisticStateExtension extends object = {}> extends HttpCrawler<JSDOMCrawlingContext, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
+ constructor(options?: JSDOMCrawlerOptions<ContextExtension, ExtendedContext, any, any, Routes, StatisticStateExtension>);
// (undocumented)
protected buildContextPipeline(): ContextPipeline<CrawlingContext<Dictionary>, InternalHttpCrawlingContext<any, any> & {
readonly window: DOMWindow;
@@ -169,7 +169,7 @@ export class JSDOMCrawler<ContextExtension = Dictionary<never>, ExtendedContext
// @public (undocumented)
export interface JSDOMCrawlerOptions<ContextExtension = Dictionary<never>, ExtendedContext extends JSDOMCrawlingContext = JSDOMCrawlingContext & ContextExtension, UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
-Routes extends Record<keyof Routes, Dictionary> = Record<string, UserData>> extends HttpCrawlerOptions<JSDOMCrawlingContext<UserData, JSONData>, ContextExtension, ExtendedContext, Routes> {
+Routes extends Record<keyof Routes, Dictionary> = Record<string, UserData>, StatisticStateExtension extends object = {}> extends HttpCrawlerOptions<JSDOMCrawlingContext<UserData, JSONData>, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
hideInternalConsole?: boolean;
runScripts?: boolean;
}
diff --git a/docs/public-api/crawlee-linkedom.api.md b/docs/public-api/crawlee-linkedom.api.md
index 44a404fac..1908f3dc5 100644
--- a/docs/public-api/crawlee-linkedom.api.md
+++ b/docs/public-api/crawlee-linkedom.api.md
@@ -34,8 +34,8 @@ export function createLinkeDOMRouter<Context extends LinkeDOMCrawlingContext = L
export function createLinkeDOMRouter<Context extends LinkeDOMCrawlingContext = LinkeDOMCrawlingContext, const Schemas extends RouteSchemas = RouteSchemas>(schemas: Schemas): RouterHandler<Context, RoutesFromSchemas<Schemas>>;
// @public
-export class LinkeDOMCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends LinkeDOMCrawlingContext = LinkeDOMCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<LinkeDOMCrawlingContext['request']>>> extends HttpCrawler<LinkeDOMCrawlingContext, ContextExtension, ExtendedContext, Routes> {
- constructor(options?: LinkeDOMCrawlerOptions<ContextExtension, ExtendedContext, any, any, Routes>);
+export class LinkeDOMCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends LinkeDOMCrawlingContext = LinkeDOMCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<LinkeDOMCrawlingContext['request']>>, StatisticStateExtension extends object = {}> extends HttpCrawler<LinkeDOMCrawlingContext, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
+ constructor(options?: LinkeDOMCrawlerOptions<ContextExtension, ExtendedContext, any, any, Routes, StatisticStateExtension>);
// (undocumented)
protected buildContextPipeline(): ContextPipeline<CrawlingContext<Dictionary>, InternalHttpCrawlingContext<any, any> & {
readonly window: Window;
@@ -52,7 +52,7 @@ export class LinkeDOMCrawler<ContextExtension = Dictionary<never>, ExtendedConte
// @public (undocumented)
export interface LinkeDOMCrawlerOptions<ContextExtension = Dictionary<never>, ExtendedContext extends LinkeDOMCrawlingContext = LinkeDOMCrawlingContext & ContextExtension, UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
-Routes extends Record<keyof Routes, Dictionary> = Record<string, UserData>> extends HttpCrawlerOptions<LinkeDOMCrawlingContext<UserData, JSONData>, ContextExtension, ExtendedContext, Routes> {
+Routes extends Record<keyof Routes, Dictionary> = Record<string, UserData>, StatisticStateExtension extends object = {}> extends HttpCrawlerOptions<LinkeDOMCrawlingContext<UserData, JSONData>, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
}
// @public (undocumented)
diff --git a/docs/public-api/crawlee-playwright.api.md b/docs/public-api/crawlee-playwright.api.md
index f43ea9079..d64461ed4 100644
--- a/docs/public-api/crawlee-playwright.api.md
+++ b/docs/public-api/crawlee-playwright.api.md
@@ -48,9 +48,6 @@ import type { RouterRoutes as RouterRoutes_2 } from '@crawlee/core';
import type { RouteSchemas } from '@crawlee/browser';
import type { RoutesFromSchemas } from '@crawlee/browser';
import type { SkippedRequestCallback } from '@crawlee/browser';
-import type { StatisticPersistedState } from '@crawlee/core';
-import { Statistics } from '@crawlee/core';
-import type { StatisticState } from '@crawlee/core';
import type { StorageTransactionView } from '@crawlee/core';
import type { UrlPatternInput } from '@crawlee/browser';
import { z } from 'zod';
@@ -71,8 +68,8 @@ interface AdaptiveHookContext extends Pick<AdaptivePlaywrightCrawlerContext, 'id
}
// @public
-export class AdaptivePlaywrightCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends AdaptivePlaywrightCrawlerContext = AdaptivePlaywrightCrawlerContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest_2<AdaptivePlaywrightCrawlerContext['request']>>> extends BasicCrawler<AdaptivePlaywrightCrawlerContext, ContextExtension, ExtendedContext, Routes> {
- constructor(options?: AdaptivePlaywrightCrawlerOptions<ContextExtension, ExtendedContext, Routes>);
+export class AdaptivePlaywrightCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends AdaptivePlaywrightCrawlerContext = AdaptivePlaywrightCrawlerContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest_2<AdaptivePlaywrightCrawlerContext['request']>>, StatisticStateExtension extends AdaptivePlaywrightCrawlerStatisticState = AdaptivePlaywrightCrawlerStatisticState> extends BasicCrawler<AdaptivePlaywrightCrawlerContext, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
+ constructor(options?: AdaptivePlaywrightCrawlerOptions<ContextExtension, ExtendedContext, Routes, StatisticStateExtension>);
// (undocumented)
protected buildContextPipeline(): ContextPipeline_2<CrawlingContext_2<Dictionary>, CrawlingContext_2<Dictionary> & {
readonly request: LoadedRequest<Request_3<Dictionary>>;
@@ -89,8 +86,6 @@ export class AdaptivePlaywrightCrawler<ContextExtension = Dictionary<never>, Ext
// (undocumented)
protected runRequestHandler(crawlingContext: CrawlingContext_2): Promise<void>;
// (undocumented)
- get stats(): AdaptivePlaywrightCrawlerStatistics;
- // (undocumented)
teardown(): Promise<void>;
}
@@ -109,7 +104,7 @@ export interface AdaptivePlaywrightCrawlerContext<UserData extends Dictionary =
}
// @public (undocumented)
-export interface AdaptivePlaywrightCrawlerOptions<ContextExtension = Dictionary<never>, ExtendedContext extends AdaptivePlaywrightCrawlerContext = AdaptivePlaywrightCrawlerContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest_2<AdaptivePlaywrightCrawlerContext['request']>>> extends Omit<BasicCrawlerOptions<AdaptivePlaywrightCrawlerContext, ContextExtension, ExtendedContext, Routes>, 'preNavigationHooks' | 'postNavigationHooks'> {
+export interface AdaptivePlaywrightCrawlerOptions<ContextExtension = Dictionary<never>, ExtendedContext extends AdaptivePlaywrightCrawlerContext = AdaptivePlaywrightCrawlerContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest_2<AdaptivePlaywrightCrawlerContext['request']>>, StatisticStateExtension extends AdaptivePlaywrightCrawlerStatisticState = AdaptivePlaywrightCrawlerStatisticState> extends Omit<BasicCrawlerOptions<AdaptivePlaywrightCrawlerContext, ContextExtension, ExtendedContext, Routes, StatisticStateExtension>, 'preNavigationHooks' | 'postNavigationHooks'> {
postNavigationHooks?: AdaptivePostNavigationHook<ContextExtension>[];
preNavigationHooks?: AdaptiveHook<ContextExtension>[];
renderingTypeDetectionRatio?: number;
@@ -119,44 +114,17 @@ export interface AdaptivePlaywrightCrawlerOptions<ContextExtension = Dictionary<
shouldPropagateError?: (error: Error, context: PlaywrightCrawlingContext) => Awaitable<boolean>;
}
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-interface AdaptivePlaywrightCrawlerPersistedStatisticState extends StatisticPersistedState {
- // (undocumented)
- browserRequestHandlerRuns?: number;
- // (undocumented)
- httpOnlyRequestHandlerRuns?: number;
- // (undocumented)
- renderingTypeMispredictions?: number;
-}
-
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-class AdaptivePlaywrightCrawlerStatistics extends Statistics {
- // (undocumented)
- protected defaultState(): AdaptivePlaywrightCrawlerStatisticState;
- // (undocumented)
- protected deserializeState(persistedState: AdaptivePlaywrightCrawlerPersistedStatisticState): AdaptivePlaywrightCrawlerStatisticState;
- // (undocumented)
- get state(): AdaptivePlaywrightCrawlerStatisticState;
- // (undocumented)
- trackBrowserRequestHandlerRun(): void;
- // (undocumented)
- trackHttpOnlyRequestHandlerRun(): void;
- // (undocumented)
- trackRenderingTypeMisprediction(): void;
-}
+// @public
+export type AdaptivePlaywrightCrawlerStatisticState = z.infer<typeof adaptiveStatisticStateSchema>;
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-interface AdaptivePlaywrightCrawlerStatisticState extends StatisticState {
- // (undocumented)
- browserRequestHandlerRuns?: number;
- // (undocumented)
- httpOnlyRequestHandlerRuns?: number;
- // (undocumented)
- renderingTypeMispredictions?: number;
-}
+// @public
+export const adaptivePlaywrightCrawlerStatisticState: {
+ deserialize: z.ZodObject<{
+ httpOnlyRequestHandlerRuns: z.ZodDefault<z.ZodNumber>;
+ browserRequestHandlerRuns: z.ZodDefault<z.ZodNumber>;
+ renderingTypeMispredictions: z.ZodDefault<z.ZodNumber>;
+ }, z.core.$strip>;
+};
// Not exported by the entry point; reachable only as a referenced type.
// @public (undocumented)
@@ -164,6 +132,14 @@ type AdaptivePostNavigationHook<ContextExtension = Dictionary<never>> = BrowserH
request: LoadedRequest<Request_3>;
}, ContextExtension>;
+// Not exported by the entry point; reachable only as a referenced type.
+// @public (undocumented)
+const adaptiveStatisticStateSchema: z.ZodObject<{
+ httpOnlyRequestHandlerRuns: z.ZodDefault<z.ZodNumber>;
+ browserRequestHandlerRuns: z.ZodDefault<z.ZodNumber>;
+ renderingTypeMispredictions: z.ZodDefault<z.ZodNumber>;
+}, z.core.$strip>;
+
// @public
function blockRequests(page: Page, options?: BlockRequestsOptions): Promise<void>;
@@ -326,10 +302,10 @@ interface PlaywrightContextUtils {
}
// @public
-export class PlaywrightCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends PlaywrightCrawlingContext = PlaywrightCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<PlaywrightCrawlingContext['request']>>> extends BrowserCrawler<Page, Response_2, {
+export class PlaywrightCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends PlaywrightCrawlingContext = PlaywrightCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<PlaywrightCrawlingContext['request']>>, StatisticStateExtension extends object = {}> extends BrowserCrawler<Page, Response_2, {
browserPlugins: [PlaywrightPlugin];
-}, LaunchOptions, PlaywrightCrawlingContext, ContextExtension, ExtendedContext, Routes> {
- constructor(options?: PlaywrightCrawlerOptions<ContextExtension, ExtendedContext, Routes>);
+}, LaunchOptions, PlaywrightCrawlingContext, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
+ constructor(options?: PlaywrightCrawlerOptions<ContextExtension, ExtendedContext, Routes, StatisticStateExtension>);
// (undocumented)
protected buildContextPipeline(): ContextPipeline<CrawlingContext<Dictionary>, BrowserCrawlingContext<Page, Response_2, Dictionary, Dictionary> & {
injectFile: (filePath: string, options?: InjectFileOptions) => Promise<unknown>;
@@ -464,9 +440,9 @@ export class PlaywrightCrawler<ContextExtension = Dictionary<never>, ExtendedCon
}
// @public (undocumented)
-export interface PlaywrightCrawlerOptions<ContextExtension = Dictionary<never>, ExtendedContext extends PlaywrightCrawlingContext = PlaywrightCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<PlaywrightCrawlingContext['request']>>> extends BrowserCrawlerOptions<Page, Response_2, PlaywrightCrawlingContext, ContextExtension, ExtendedContext, {
+export interface PlaywrightCrawlerOptions<ContextExtension = Dictionary<never>, ExtendedContext extends PlaywrightCrawlingContext = PlaywrightCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<PlaywrightCrawlingContext['request']>>, StatisticStateExtension extends object = {}> extends BrowserCrawlerOptions<Page, Response_2, PlaywrightCrawlingContext, ContextExtension, ExtendedContext, {
browserPlugins: [PlaywrightPlugin];
-}, Routes> {
+}, Routes, StatisticStateExtension> {
launchContext?: PlaywrightLaunchContext;
postNavigationHooks?: BrowserHook<PlaywrightCrawlingContext<GetUserDataFromRequest<ExtendedContext['request']>>, ContextExtension>[];
preNavigationHooks?: BrowserHook<PlaywrightCrawlingContext<GetUserDataFromRequest<ExtendedContext['request']>>, ContextExtension>[];
diff --git a/docs/public-api/crawlee-puppeteer.api.md b/docs/public-api/crawlee-puppeteer.api.md
index e13eee255..92cbf390b 100644
--- a/docs/public-api/crawlee-puppeteer.api.md
+++ b/docs/public-api/crawlee-puppeteer.api.md
@@ -172,10 +172,10 @@ interface PuppeteerContextUtils {
}
// @public
-export class PuppeteerCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends PuppeteerCrawlingContext = PuppeteerCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<PuppeteerCrawlingContext['request']>>> extends BrowserCrawler<Page, HTTPResponse, {
+export class PuppeteerCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends PuppeteerCrawlingContext = PuppeteerCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<PuppeteerCrawlingContext['request']>>, StatisticStateExtension extends object = {}> extends BrowserCrawler<Page, HTTPResponse, {
browserPlugins: [PuppeteerPlugin];
-}, LaunchOptions, PuppeteerCrawlingContext, ContextExtension, ExtendedContext, Routes> {
- constructor(options?: PuppeteerCrawlerOptions<ContextExtension, ExtendedContext, Routes>);
+}, LaunchOptions, PuppeteerCrawlingContext, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
+ constructor(options?: PuppeteerCrawlerOptions<ContextExtension, ExtendedContext, Routes, StatisticStateExtension>);
// (undocumented)
protected buildContextPipeline(): ContextPipeline<CrawlingContext<Dictionary>, BrowserCrawlingContext<Page, HTTPResponse, Dictionary, Dictionary> & {
injectFile: (filePath: string, options?: InjectFileOptions) => Promise<unknown>;
@@ -308,9 +308,9 @@ export class PuppeteerCrawler<ContextExtension = Dictionary<never>, ExtendedCont
}
// @public (undocumented)
-export interface PuppeteerCrawlerOptions<ContextExtension = Dictionary<never>, ExtendedContext extends PuppeteerCrawlingContext = PuppeteerCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<PuppeteerCrawlingContext['request']>>> extends BrowserCrawlerOptions<Page, HTTPResponse, PuppeteerCrawlingContext, ContextExtension, ExtendedContext, {
+export interface PuppeteerCrawlerOptions<ContextExtension = Dictionary<never>, ExtendedContext extends PuppeteerCrawlingContext = PuppeteerCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<PuppeteerCrawlingContext['request']>>, StatisticStateExtension extends object = {}> extends BrowserCrawlerOptions<Page, HTTPResponse, PuppeteerCrawlingContext, ContextExtension, ExtendedContext, {
browserPlugins: [PuppeteerPlugin];
-}, Routes> {
+}, Routes, StatisticStateExtension> {
launchContext?: PuppeteerLaunchContext;
postNavigationHooks?: BrowserHook<PuppeteerCrawlingContext<GetUserDataFromRequest<ExtendedContext['request']>>, ContextExtension>[];
preNavigationHooks?: BrowserHook<PuppeteerCrawlingContext<GetUserDataFromRequest<ExtendedContext['request']>>, ContextExtension>[];
diff --git a/docs/public-api/crawlee-stagehand.api.md b/docs/public-api/crawlee-stagehand.api.md
index 0c7cccade..02c0224b3 100644
--- a/docs/public-api/crawlee-stagehand.api.md
+++ b/docs/public-api/crawlee-stagehand.api.md
@@ -73,10 +73,10 @@ export { ObserveOptions }
export { Stagehand }
// @public
-export class StagehandCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends StagehandCrawlingContext = StagehandCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<StagehandCrawlingContext['request']>>> extends BrowserCrawler<StagehandPage, Response_2, {
+export class StagehandCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends StagehandCrawlingContext = StagehandCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<StagehandCrawlingContext['request']>>, StatisticStateExtension extends object = {}> extends BrowserCrawler<StagehandPage, Response_2, {
browserPlugins: [StagehandPlugin];
-}, LaunchOptions, StagehandCrawlingContext, ContextExtension, ExtendedContext, Routes> {
- constructor(options?: StagehandCrawlerOptions<ContextExtension, ExtendedContext, Routes>);
+}, LaunchOptions, StagehandCrawlingContext, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
+ constructor(options?: StagehandCrawlerOptions<ContextExtension, ExtendedContext, Routes, StatisticStateExtension>);
// (undocumented)
protected buildContextPipeline(): ContextPipeline<CrawlingContext, StagehandCrawlingContext>;
protected navigationHandler(crawlingContext: StagehandCrawlingContext, gotoOptions: StagehandGotoOptions): Promise<Response_2 | null>;
@@ -197,9 +197,9 @@ export class StagehandCrawler<ContextExtension = Dictionary<never>, ExtendedCont
}
// @public
-export interface StagehandCrawlerOptions<ContextExtension = Dictionary<never>, ExtendedContext extends StagehandCrawlingContext = StagehandCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<StagehandCrawlingContext['request']>>> extends BrowserCrawlerOptions<StagehandPage, Response_2, StagehandCrawlingContext, ContextExtension, ExtendedContext, {
+export interface StagehandCrawlerOptions<ContextExtension = Dictionary<never>, ExtendedContext extends StagehandCrawlingContext = StagehandCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<StagehandCrawlingContext['request']>>, StatisticStateExtension extends object = {}> extends BrowserCrawlerOptions<StagehandPage, Response_2, StagehandCrawlingContext, ContextExtension, ExtendedContext, {
browserPlugins: [StagehandPlugin];
-}, Routes> {
+}, Routes, StatisticStateExtension> {
launchContext?: StagehandLaunchContext;
postNavigationHooks?: BrowserHook<StagehandCrawlingContext<GetUserDataFromRequest<ExtendedContext['request']>>, ContextExtension>[];
preNavigationHooks?: BrowserHook<StagehandCrawlingContext<GetUserDataFromRequest<ExtendedContext['request']>>, ContextExtension>[];
diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.md
index f898f966a..291bdc828 100644
--- a/docs/upgrading/upgrading_v4.md
+++ b/docs/upgrading/upgrading_v4.md
@@ -1205,7 +1205,7 @@ If you called the standalone `playwrightUtils.handleCloudflareChallenge(page, ur
## Only if you customize crawler statistics
-Applies when you passed `statisticsOptions` to a crawler, or subclassed `Statistics`.
+Applies when you passed `statisticsOptions` to a crawler, subclassed `Statistics`, or passed type arguments to `BrowserCrawler`/`BrowserCrawlerOptions`.
### `statisticsOptions` is replaced by a `statistics` instance
@@ -1221,7 +1221,7 @@ const crawler = new BasicCrawler({
});
```
-Omit the option and the crawler builds its own default, exactly as before. A supplied instance is treated as borrowed: the crawler records into it and drives its capture lifecycle for the run, but never `reset()`s it between `run()` calls — so a preconfigured instance keeps whatever state it was handed. This is also the supported way to plug in a custom `Statistics` subclass that tracks extra fields (superseding the previous subclass-and-reassign pattern).
+Omit the option and the crawler builds its own default, exactly as before. A supplied instance is treated as borrowed: the crawler records into it and drives its capture lifecycle for the run, but never `reset()`s it between `run()` calls — so a preconfigured instance keeps whatever state it was handed.
The option accepts the built-in `Statistics` or any object implementing the new `IStatistics` interface, so a fully custom statistics backend can be plugged in without subclassing. The crawler exposes it as `crawler.stats` typed as `IStatistics`.
@@ -1231,6 +1231,19 @@ The option accepts the built-in `Statistics` or any object implementing the new
A persisted record is also validated on load now. One that does not match the expected shape is discarded whole, with a warning, and the statistics start from scratch — where v3 would copy the malformed values into the live state and let them corrupt every later increment.
+### Subclassing `Statistics` to track extra fields is replaced by the `stateExtension` option
+
+`defaultState()`, `serializeState()`, `deserializeState()` and `persistStateKey` were `protected` and are now private, so a subclass can no longer override them. Declare the extra fields via the new `stateExtension` option instead — `{ defaultState, deserialize, serialize }`, the same trio `RecoverableState` takes, scoped to the custom fields. See the [Custom statistics fields](../guides/custom-statistics) guide.
+
+A consequence of the hooks going away: the persisted record is now validated strictly, and keys that are neither built-in nor declared in `stateExtension` are dropped rather than written back. `calculate()` is still public and still an override point.
+
+The custom field types reach `crawler.stats.state` through a new trailing `StatisticStateExtension` type parameter on the crawler classes and their options. It defaults to `{}`, so existing type arguments keep working — except on `BrowserCrawler` and `BrowserCrawlerOptions`, where it was inserted after `Routes` and shifts the trailing internal parameters (`GoToOptions`, `__BrowserPlugins`, …). Adjust any explicit type arguments you passed to those two.
+
+`AdaptivePlaywrightCrawler` now uses this mechanism for its own extra fields, with two consequences:
+
+- `httpOnlyRequestHandlerRuns`, `browserRequestHandlerRuns` and `renderingTypeMispredictions` were typed as optional and are now always present. Reading them no longer needs a `?? 0`.
+- The `statistics` option used to throw for this crawler; it now accepts any `IStatistics<AdaptivePlaywrightCrawlerStatisticState>`. Build one by extending the exported `adaptivePlaywrightCrawlerStatisticState`.
+
## Only if you wrote a custom HTTP client or used `got-scraping` directly
Applies when you implemented `BaseHttpClient` yourself, or imported `gotScraping` from `@crawlee/utils`.
diff --git a/packages/basic-crawler/src/internals/basic-crawler.ts b/packages/basic-crawler/src/internals/basic-crawler.ts
index 7964e3d6a..8198f8973 100644
--- a/packages/basic-crawler/src/internals/basic-crawler.ts
+++ b/packages/basic-crawler/src/internals/basic-crawler.ts
@@ -232,6 +232,7 @@ export interface BasicCrawlerOptions<
ContextExtension = Dictionary<never>,
ExtendedContext extends Context = Context & ContextExtension,
Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>,
+ StatisticStateExtension extends object = {},
> {
/**
* User-provided function that performs the logic of the crawler. It is called for each URL to crawl.
@@ -505,10 +506,28 @@ export interface BasicCrawlerOptions<
/**
* A preconfigured statistics instance. When provided, the crawler records into it instead of building its own and
- * will not `reset()` it between `run()` calls. Accepts the built-in {@apilink Statistics} (subclass it to track
- * extra fields) or any object implementing {@apilink IStatistics}.
+ * will not `reset()` it between `run()` calls. Accepts the built-in {@apilink Statistics} or any object
+ * implementing {@apilink IStatistics}.
+ *
+ * Custom fields declared via {@apilink StatisticsOptions.stateExtension|`stateExtension`} are carried over to
+ * {@apilink BasicCrawler.stats|`crawler.stats.state`}:
+ *
+ * ```ts
+ * const statistics = new Statistics({ stateExtension: { defaultState: { productsFound: 0 } } });
+ *
+ * const crawler = new BasicCrawler({
+ * statistics,
+ * requestHandler: async () => {
+ * statistics.state.productsFound++;
+ * },
+ * });
+ *
+ * await crawler.run();
+ * // the custom fields are typed on `crawler.stats` too
+ * console.log(crawler.stats.state.productsFound);
+ * ```
*/
- statistics?: IStatistics;
+ statistics?: IStatistics<StatisticStateExtension>;
/**
* HTTP client implementation for the `sendRequest` context helper and for plain HTTP crawling.
@@ -678,6 +697,7 @@ export class BasicCrawler<
ContextExtension = Dictionary<never>,
ExtendedContext extends Context = Context & ContextExtension,
Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>,
+ StatisticStateExtension extends object = {},
> {
protected static readonly CRAWLEE_STATE_KEY = 'CRAWLEE_STATE';
@@ -696,13 +716,13 @@ export class BasicCrawler<
static #useStateAnonymousIndices = new Set<number>();
/** Backs the {@apilink BasicCrawler.stats|`stats`} getter. */
- #statsDep: OwnedOrInjected<IStatistics, Statistics>;
+ #statsDep: OwnedOrInjected<IStatistics<StatisticStateExtension>, Statistics<StatisticStateExtension>>;
/**
* The statistics instance collecting the crawler's run statistics - either the injected `statistics` option or a
* crawler-built default. Typed as {@apilink IStatistics} so custom implementations can be plugged in.
*/
- get stats(): IStatistics {
+ get stats(): IStatistics<StatisticStateExtension> {
return this.#statsDep.value;
}
@@ -935,7 +955,7 @@ export class BasicCrawler<
* All `BasicCrawler` parameters are passed via an options object.
*/
constructor(
- options: BasicCrawlerOptions<Context, ContextExtension, ExtendedContext, Routes> &
+ options: BasicCrawlerOptions<Context, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> &
RequireContextPipeline<CrawlingContext, Context> = {} as any, // cast because the constructor logic handles missing `contextPipelineBuilder` - the type is just for DX
) {
const parsedOptions = parseArgument(options, BasicCrawler.optionsSchema, 'BasicCrawlerOptions');
@@ -1110,14 +1130,20 @@ export class BasicCrawler<
this.maxRequestRetries = maxRequestRetries;
this.maxCrawlDepth = maxCrawlDepth;
this.#sameDomainDelaySecs = sameDomainDelaySecs;
- this.#statsDep = OwnedOrInjected.resolve<IStatistics, Statistics>(
+ this.#statsDep = OwnedOrInjected.resolve<
+ IStatistics<StatisticStateExtension>,
+ Statistics<StatisticStateExtension>
+ >(
statistics,
+ // A crawler-built default tracks the built-in fields only. A non-empty `StatisticStateExtension` can
+ // only be satisfied by an injected instance carrying the matching `state`, so this branch does
+ // not run in that case - hence the cast.
() =>
new Statistics({
logMessage: `${this.constructor.name} request statistics:`,
log: this.log,
id: this.identity.id,
- }),
+ }) as Statistics<StatisticStateExtension>,
);
if (sessionPool && proxyConfiguration) {
diff --git a/packages/browser-crawler/src/internals/browser-crawler.ts b/packages/browser-crawler/src/internals/browser-crawler.ts
index b58dd67a3..42558e4a7 100644
--- a/packages/browser-crawler/src/internals/browser-crawler.ts
+++ b/packages/browser-crawler/src/internals/browser-crawler.ts
@@ -121,11 +121,12 @@ export interface BrowserCrawlerOptions<
ExtendedContext extends Context = Context & ContextExtension,
InternalBrowserPoolOptions extends BrowserPoolOptions = BrowserPoolOptions,
Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>,
+ StatisticStateExtension extends object = {},
__BrowserPlugins extends BrowserPlugin[] = InferBrowserPluginArray<InternalBrowserPoolOptions['browserPlugins']>,
__BrowserControllerReturn extends BrowserController = ReturnType<__BrowserPlugins[number]['createController']>,
__LaunchContextReturn extends LaunchContext = ReturnType<__BrowserPlugins[number]['createLaunchContext']>,
> extends Omit<
- BasicCrawlerOptions<Context, ContextExtension, ExtendedContext>,
+ BasicCrawlerOptions<Context, ContextExtension, ExtendedContext, Routes, StatisticStateExtension>,
// Overridden with browser context
'requestHandler' | 'failedRequestHandler' | 'errorHandler'
> {
@@ -359,8 +360,9 @@ export abstract class BrowserCrawler<
ContextExtension = Dictionary<never>,
ExtendedContext extends Context = Context & ContextExtension,
Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>,
+ StatisticStateExtension extends object = {},
GoToOptions extends Dictionary = Dictionary,
-> extends BasicCrawler<Context, ContextExtension, ExtendedContext, Routes> {
+> extends BasicCrawler<Context, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
/** Backs the {@apilink BrowserCrawler.browserPool|`browserPool`} getter. */
#browserPoolDep: OwnedOrInjected<IBrowserPool<Page>, OwnedBrowserPool<Page>>;
@@ -408,7 +410,18 @@ export abstract class BrowserCrawler<
* All `BrowserCrawler` parameters are passed via an options object.
*/
protected constructor(
- options: BrowserCrawlerOptions<Page, Response, Context, ContextExtension, ExtendedContext> & {
+ options: BrowserCrawlerOptions<
+ Page,
+ Response,
+ Context,
+ ContextExtension,
+ ExtendedContext,
+ // Left at the default rather than forwarding `InternalBrowserPoolOptions` - narrowing the pool options
+ // here would reject subclasses that declare their own options without pinning the pool type.
+ BrowserPoolOptions,
+ Routes,
+ StatisticStateExtension
+ > & {
contextPipelineBuilder: () => ContextPipeline<CrawlingContext, Context>;
},
) {
diff --git a/packages/cheerio-crawler/src/internals/cheerio-crawler.ts b/packages/cheerio-crawler/src/internals/cheerio-crawler.ts
index f2ab7eb1b..f74ef217d 100644
--- a/packages/cheerio-crawler/src/internals/cheerio-crawler.ts
+++ b/packages/cheerio-crawler/src/internals/cheerio-crawler.ts
@@ -39,7 +39,14 @@ export interface CheerioCrawlerOptions<
UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
Routes extends Record<keyof Routes, Dictionary> = Record<string, UserData>,
-> extends HttpCrawlerOptions<CheerioCrawlingContext<UserData, JSONData>, ContextExtension, ExtendedContext, Routes> {}
+ StatisticStateExtension extends object = {},
+> extends HttpCrawlerOptions<
+ CheerioCrawlingContext<UserData, JSONData>,
+ ContextExtension,
+ ExtendedContext,
+ Routes,
+ StatisticStateExtension
+> {}
export type CheerioHook<
UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
@@ -193,11 +200,14 @@ export class CheerioCrawler<
string,
GetUserDataFromRequest<CheerioCrawlingContext['request']>
>,
-> extends HttpCrawler<CheerioCrawlingContext, ContextExtension, ExtendedContext, Routes> {
+ StatisticStateExtension extends object = {},
+> extends HttpCrawler<CheerioCrawlingContext, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
/**
* All `CheerioCrawler` parameters are passed via an options object.
*/
- constructor(options?: CheerioCrawlerOptions<ContextExtension, ExtendedContext, any, any, Routes>) {
+ constructor(
+ options?: CheerioCrawlerOptions<ContextExtension, ExtendedContext, any, any, Routes, StatisticStateExtension>,
+ ) {
const { contextPipelineBuilder, ...rest } = options ?? {};
super({
diff --git a/packages/core/package.json b/packages/core/package.json
index ec797c596..17316b975 100644
--- a/packages/core/package.json
+++ b/packages/core/package.json
@@ -57,6 +57,7 @@
"@crawlee/types": "workspace:*",
"@crawlee/utils": "workspace:*",
"@sapphire/async-queue": "^1.5.5",
+ "@standard-schema/spec": "^1.0.0",
"@vladfrangu/async_event_emitter": "^2.4.6",
"content-type": "^1.0.5",
"csv-stringify": "^6.5.2",
diff --git a/packages/core/src/crawlers/statistics.ts b/packages/core/src/crawlers/statistics.ts
index b9e51e5de..e18a0e496 100644
--- a/packages/core/src/crawlers/statistics.ts
+++ b/packages/core/src/crawlers/statistics.ts
@@ -1,7 +1,8 @@
import { z } from 'zod';
import type { CrawleeLogger } from '../log.js';
-import { RecoverableState } from '../recoverable_state.js';
+import type { SyncStateConversion } from '../recoverable_state.js';
+import { convertStateSync, RecoverableState } from '../recoverable_state.js';
import { serviceLocator } from '../service_locator.js';
import { KeyValueStore } from '../storages/key_value_store.js';
import { parseArgument, schemas, validators } from '../validators.js';
@@ -33,6 +34,7 @@ const statisticsOptionsSchema = z.strictObject({
persistenceOptions: schemas.anyObject.default(() => ({ enable: true })),
saveErrorSnapshots: z.boolean().default(false),
id: z.union([schemas.anyNumber, z.string()]).optional(),
+ stateExtension: schemas.anyObject.default(() => ({})),
});
const errorTrackerConfig = {
@@ -52,41 +54,40 @@ const errorTrackerConfig = {
* `null`. Both {@apilink Statistics.serializeState} and {@apilink Statistics.deserializeState} run through this,
* which is what keeps them describing the same record.
*
- * Nothing is optional on purpose: `serializeState` has always written every field, so a record missing one is not
- * one of ours and is discarded whole rather than partially trusted - a counter restored as a string would poison
- * every later increment.
+ * Nothing is optional on purpose: the record has always carried every field, so one missing a field is not one of
+ * ours and is discarded whole rather than partially trusted - a counter restored as a string would poison every
+ * later increment.
+ *
+ * Custom fields are not this schema's business either way: they are added to the record after the encode and
+ * validated by their own conversion on the way back, so the keys it does not know about are simply dropped here.
*/
-const persistedStatisticState = z
- .object({
- requestsFinished: z.number(),
- requestsFailed: z.number(),
- requestsRetries: z.number(),
- requestsFailedPerMinute: z.number().nullable(),
- requestsFinishedPerMinute: z.number().nullable(),
- requestMinDurationMillis: z.number().nullable(),
- requestMaxDurationMillis: z.number(),
- requestTotalFailedDurationMillis: z.number(),
- requestTotalFinishedDurationMillis: z.number(),
- crawlerStartedAt: z.string().nullable(),
- crawlerFinishedAt: z.string().nullable(),
- statsPersistedAt: z.string(),
- crawlerRuntimeMillis: z.number(),
- crawlerLastStartTimestamp: z.number(),
- // A retry count that never occurred leaves a hole in the live histogram, written out as a `null`. We
- // once saw a record whose histogram was not an array at all and crashed the crawler on load.
- requestRetryHistogram: z.array(z.number().nullable()),
- statsId: z.string(),
- requestAvgFailedDurationMillis: z.number().nullable(),
- requestAvgFinishedDurationMillis: z.number().nullable(),
- requestTotalDurationMillis: z.number(),
- requestsTotal: z.number(),
- requestsWithStatusCode: z.record(z.string(), z.number()),
- errors: z.record(z.string(), z.unknown()),
- retryErrors: z.record(z.string(), z.unknown()),
- })
- // A subclass tracking extra fields spreads them into the record; they are none of this schema's business,
- // but they must not be dropped on the way through it.
- .catchall(z.unknown());
+const persistedStatisticState = z.object({
+ requestsFinished: z.number(),
+ requestsFailed: z.number(),
+ requestsRetries: z.number(),
+ requestsFailedPerMinute: z.number().nullable(),
+ requestsFinishedPerMinute: z.number().nullable(),
+ requestMinDurationMillis: z.number().nullable(),
+ requestMaxDurationMillis: z.number(),
+ requestTotalFailedDurationMillis: z.number(),
+ requestTotalFinishedDurationMillis: z.number(),
+ crawlerStartedAt: z.string().nullable(),
+ crawlerFinishedAt: z.string().nullable(),
+ statsPersistedAt: z.string(),
+ crawlerRuntimeMillis: z.number(),
+ crawlerLastStartTimestamp: z.number(),
+ // A retry count that never occurred leaves a hole in the live histogram, written out as a `null`. We
+ // once saw a record whose histogram was not an array at all and crashed the crawler on load.
+ requestRetryHistogram: z.array(z.number().nullable()),
+ statsId: z.string(),
+ requestAvgFailedDurationMillis: z.number().nullable(),
+ requestAvgFinishedDurationMillis: z.number().nullable(),
+ requestTotalDurationMillis: z.number(),
+ requestsTotal: z.number(),
+ requestsWithStatusCode: z.record(z.string(), z.number()),
+ errors: z.record(z.string(), z.unknown()),
+ retryErrors: z.record(z.string(), z.unknown()),
+});
/** `Infinity` is what the statistics use for "nothing to average yet"; JSON has only `null` for it. */
function finiteOrNull(value: number): number | null {
@@ -98,8 +99,7 @@ function finiteOrNull(value: number): number | null {
*
* Built per instance rather than kept as a constant because a record carries three things the state does not: the
* instance `id`, the derived aggregates of the overridable {@apilink Statistics.calculate}, and - on the way back -
- * the fields that are rebuilt from {@apilink Statistics.defaultState} rather than restored, the error trackers
- * among them.
+ * the fields that are rebuilt from the instance's default state rather than restored, the error trackers among them.
*
* The model side is deliberately opaque: zod rebuilds what it validates, and `state.errors` has to stay the very
* object the error trackers write into, not a copy of it.
@@ -182,8 +182,11 @@ export interface PersistenceOptions {
/**
* The statistics surface a crawler depends on: recording per-request outcomes, tracking errors, and driving the
- * capture lifecycle for a run. Injected via the crawler's `statistics` option, so a custom implementation (or a
- * {@apilink Statistics} subclass tracking extra fields) can be plugged in without subclassing the crawler.
+ * capture lifecycle for a run. Injected via the crawler's `statistics` option, so a custom implementation can be
+ * plugged in without subclassing the crawler.
+ *
+ * `StateExtension` describes the custom fields tracked alongside the built-in {@apilink StatisticState} ones - see
+ * {@apilink StatisticsOptions.stateExtension}.
*
* The owned-only mutators the crawler uses to *own* a default it built - `reset()`/`resetStore()` - are deliberately
* absent: an injected instance is borrowed, and the crawler never wipes it. Those live on the concrete
@@ -191,7 +194,7 @@ export interface PersistenceOptions {
*
* @category Crawlers
*/
-export interface IStatistics {
+export interface IStatistics<StateExtension extends object = {}> {
/** Tracker for errors on the final retry of a request. */
readonly errorTracker: ErrorTracker;
@@ -199,7 +202,7 @@ export interface IStatistics {
readonly errorTrackerRetry: ErrorTracker;
/** The live statistics state the crawler reads for status messages and the final summary. */
- readonly state: StatisticState;
+ readonly state: StatisticState & StateExtension;
/** Retries histogram - index `i` holds the number of requests that finished after `i` retries. */
readonly requestRetryHistogram: number[];
@@ -267,9 +270,15 @@ export interface CalculatedStatistics {
* under the key `CRAWLEE_CRAWLER_STATISTICS_*`, persists between
* migrations and abort/resurrect
*
+ * Custom fields are tracked by passing {@apilink StatisticsOptions.stateExtension|`stateExtension`} - the extra fields are then part
+ * of {@apilink Statistics.state|`state`}, persisted and restored along with the built-in ones.
+ *
* @category Crawlers
*/
-export class Statistics implements IStatistics {
+export class Statistics<
+ StateExtension extends object = {},
+ PersistedStateExtension extends object = StateExtension,
+> implements IStatistics<StateExtension> {
// kept as TS-private: statistics tests read the static counter directly
private static id = 0;
@@ -288,9 +297,15 @@ export class Statistics implements IStatistics {
*/
readonly id: string;
- protected readonly persistStateKey: string;
+ readonly #persistStateKey: string;
readonly #stateCodec: ReturnType<typeof buildStatisticStateCodec>;
- readonly #recoverableState: RecoverableState<StatisticState, StatisticPersistedState>;
+ readonly #recoverableState: RecoverableState<
+ StatisticState & StateExtension,
+ StatisticPersistedState & PersistedStateExtension
+ >;
+ readonly #stateExtension: StatisticStateExtensionOptions<StateExtension, PersistedStateExtension>;
+ readonly #defaultStateExtension: () => StateExtension;
+ readonly #stateExtensionKeys: readonly (keyof StateExtension)[];
#logIntervalMillis: number;
#logMessage: string;
#requestsInProgress = new Map<number | string, Job>();
@@ -300,7 +315,7 @@ export class Statistics implements IStatistics {
/**
* Current statistic state used for doing calculations on {@apilink Statistics.calculate} calls
*/
- get state(): StatisticState {
+ get state(): StatisticState & StateExtension {
return this.#recoverableState.currentValue;
}
@@ -313,36 +328,59 @@ export class Statistics implements IStatistics {
/**
* Construct a statistics instance to pass to a crawler via its `statistics` option, e.g. to preconfigure
- * persistence or error snapshots, share it across sequential runs, or subclass it to track extra fields.
+ * persistence or error snapshots, share it across sequential runs, or track extra fields via `state`.
*/
- constructor(options: StatisticsOptions = {}) {
- const { logIntervalSecs, logMessage, log, keyValueStore, persistenceOptions, saveErrorSnapshots, id } =
- parseArgument(options, statisticsOptionsSchema);
+ constructor(options: StatisticsOptions<StateExtension, PersistedStateExtension> = {}) {
+ const {
+ logIntervalSecs,
+ logMessage,
+ log,
+ keyValueStore,
+ persistenceOptions,
+ saveErrorSnapshots,
+ id,
+ stateExtension,
+ } = parseArgument(options, statisticsOptionsSchema);
this.id = id ?? String(Statistics.id++);
- this.persistStateKey = `CRAWLEE_CRAWLER_STATISTICS_${this.id}`;
+ this.#persistStateKey = `CRAWLEE_CRAWLER_STATISTICS_${this.id}`;
this.log = (log ?? serviceLocator.getLogger()).child({ prefix: 'Statistics' });
this.errorTracker = new ErrorTracker({ ...errorTrackerConfig, saveErrorSnapshots });
this.errorTrackerRetry = new ErrorTracker({ ...errorTrackerConfig, saveErrorSnapshots });
this.#logIntervalMillis = logIntervalSecs * 1000;
this.#logMessage = logMessage;
+ this.#stateExtension = stateExtension as StatisticStateExtensionOptions<
+ StateExtension,
+ PersistedStateExtension
+ >;
+ this.#defaultStateExtension = this.#resolveDefaultStateExtension(this.#stateExtension);
+ this.#stateExtensionKeys = Object.keys(this.#defaultStateExtension()) as (keyof StateExtension)[];
+
+ for (const key of this.#stateExtensionKeys) {
+ if ((key as string) in this.#builtInDefaultState()) {
+ throw new Error(
+ `The custom statistics field \`${String(key)}\` collides with a built-in one - it would shadow ` +
+ 'the value the crawler tracks. Rename it in `stateExtension`.',
+ );
+ }
+ }
- // Late-bound on purpose - both hooks are override points, and a subclass's must be the ones that run.
+ // `calculate()` is late-bound on purpose - it is an override point, and a subclass's must be the one that runs.
this.#stateCodec = buildStatisticStateCodec({
statsId: this.id,
- defaultState: () => this.defaultState(),
+ defaultState: () => this.#defaultState(),
calculate: () => this.calculate(),
});
this.#recoverableState = new RecoverableState({
- persistStateKey: this.persistStateKey,
+ persistStateKey: this.#persistStateKey,
persistenceEnabled: persistenceOptions.enable,
keyValueStore,
logger: this.log,
- defaultState: () => this.defaultState(),
- serialize: (state) => this.serializeState(state),
- deserialize: (persistedState) => this.deserializeState(persistedState),
+ defaultState: () => this.#defaultState(),
+ serialize: (state) => this.#serializeState(state),
+ deserialize: (persistedState) => this.#deserializeState(persistedState),
});
// initialize by "resetting"
@@ -361,12 +399,54 @@ export class Statistics implements IStatistics {
this.#requestsInProgress.clear();
}
+ /** The pristine state a new instance starts with and {@apilink Statistics.reset} restores. */
+ #defaultState(): StatisticState & StateExtension {
+ return {
+ ...this.#builtInDefaultState(),
+ ...this.#defaultStateExtension(),
+ };
+ }
+
/**
- * The pristine state a new instance starts with and {@apilink Statistics.reset} restores.
+ * The factory behind the custom half of the default state - it has to hand out a fresh object every time, or a
+ * `reset()` would write through to the defaults of the next one.
*
- * A subclass tracking extra fields declares their initial values here.
+ * With no `defaultState` given, the defaults are what `deserialize` makes of an empty record. That keeps a
+ * single declaration of the custom fields - a schema with a `.default()` per field is enough - and the defaults
+ * cannot then disagree with the conversion that has to accept them back.
*/
- protected defaultState(): StatisticState {
+ #resolveDefaultStateExtension(
+ options: StatisticStateExtensionOptions<StateExtension, PersistedStateExtension>,
+ ): () => StateExtension {
+ const { defaultState, deserialize } = options;
+
+ if (typeof defaultState === 'function') {
+ return defaultState;
+ }
+
+ if (defaultState !== undefined) {
+ return () => structuredClone(defaultState);
+ }
+
+ if (deserialize === undefined) {
+ return () => ({}) as StateExtension;
+ }
+
+ return () => {
+ try {
+ return convertStateSync(deserialize, {}, this.#persistStateKey);
+ } catch (error) {
+ throw new Error(
+ 'Could not derive the default values of the custom statistics fields from `stateExtension.deserialize` - ' +
+ 'give every field a default, or declare `stateExtension.defaultState` explicitly.',
+ { cause: error },
+ );
+ }
+ };
+ }
+
+ /** The built-in half of {@apilink Statistics.defaultState}, before any custom fields are merged over it. */
+ #builtInDefaultState(): StatisticState {
return {
requestsFinished: 0,
requestsFailed: 0,
@@ -541,34 +621,78 @@ export class Statistics implements IStatistics {
await this.#recoverableState
.persistState()
.catch((error) =>
- this.log.warning(`Failed to persist the statistics to ${this.persistStateKey}`, { error }),
+ this.log.warning(`Failed to persist the statistics to ${this.#persistStateKey}`, { error }),
);
}
- /**
- * Rebuilds the state from a persisted record.
- *
- * A subclass tracking extra fields restores them here, on top of the result of `super.deserializeState()`.
- */
- protected deserializeState(persistedState: StatisticPersistedState): StatisticState {
- // The cast is the index signature the `catchall` puts on the schema and an interface cannot have - the
- // record is an unvalidated blob off the key-value store either way, which is what the decode is for.
- const restored = z.safeDecode(this.#stateCodec, persistedState as z.input<typeof persistedStatisticState>);
+ /** Rebuilds the state from a persisted record. */
+ #deserializeState(
+ persistedState: StatisticPersistedState & Partial<PersistedStateExtension>,
+ ): StatisticState & StateExtension {
+ // The cast covers the custom fields, whose type is open here - and the record is an unvalidated blob off the
+ // key-value store either way, which is what the decode is for. Their keys are not in the schema, so the
+ // decode drops them; `#restoreStateExtension` is what brings them back.
+ const restored = z.safeDecode(
+ this.#stateCodec,
+ persistedState as unknown as z.input<typeof persistedStatisticState>,
+ );
if (!restored.success) {
// Statistics are bookkeeping - a record that cannot be made sense of is worth a warning and a fresh
// start, not a failed crawl.
this.log.warning('Received invalid state from Key-value store, starting the statistics from scratch.', {
- persistStateKey: this.persistStateKey,
+ persistStateKey: this.#persistStateKey,
issues: restored.error.issues,
});
- return this.defaultState();
+ return this.#defaultState();
+ }
+
+ this.log.debug('Recreating state from KeyValueStore', { persistStateKey: this.#persistStateKey });
+
+ return { ...this.#defaultState(), ...restored.data, ...this.#restoreStateExtension(persistedState) };
+ }
+
+ /**
+ * The custom {@apilink StatisticsOptions.stateExtension|`stateExtension`} fields as they were persisted - the codec only rebuilds
+ * the built-in ones.
+ *
+ * Given a `deserialize`, the whole record goes through it, so a field the record does not carry - one declared
+ * after the record was written - comes back as whatever default the conversion gives it. Without one, the
+ * declared keys are copied over as they were persisted and a missing one keeps the default it already has.
+ */
+ #restoreStateExtension(persistedState: Partial<PersistedStateExtension>): Partial<StateExtension> {
+ const { deserialize } = this.#stateExtension;
+
+ if (deserialize === undefined) {
+ const restored: Partial<StateExtension> = {};
+
+ for (const key of this.#stateExtensionKeys) {
+ // The record is an unvalidated blob and there is no conversion to check it with, so this is the
+ // caller's word for it - the state ends up holding whatever was written. The two halves only line
+ // up at all because the fields are persisted as they are without a `serialize`.
+ const persistedValue = (persistedState as unknown as Partial<StateExtension>)[key];
+
+ if (persistedValue !== undefined) {
+ restored[key] = persistedValue;
+ }
+ }
+
+ return restored;
}
- this.log.debug('Recreating state from KeyValueStore', { persistStateKey: this.persistStateKey });
+ try {
+ return convertStateSync(deserialize, persistedState, this.#persistStateKey);
+ } catch (error) {
+ // Same policy as a built-in field that cannot be made sense of, but scoped to the custom ones - a
+ // corrupt counter of your own is no reason to throw away the crawler's.
+ this.log.warning(
+ 'Received invalid custom statistics fields from Key-value store, starting those from scratch.',
+ { persistStateKey: this.#persistStateKey, error },
+ );
- return { ...this.defaultState(), ...restored.data };
+ return {};
+ }
}
#stopLogging(): void {
@@ -582,23 +706,57 @@ export class Statistics implements IStatistics {
* Builds the record written to the key value store, merging in the derived aggregates so that a consumer
* reading the record does not have to reconstruct them.
*/
- protected serializeState(state: StatisticState): StatisticPersistedState {
- return z.encode(this.#stateCodec, state);
+ #serializeState(state: StatisticState & StateExtension): StatisticPersistedState & PersistedStateExtension {
+ const { builtIn, extension } = this.#splitState(state);
+
+ return {
+ ...z.encode(this.#stateCodec, builtIn),
+ ...this.#serializeStateExtension(extension),
+ } as StatisticPersistedState & PersistedStateExtension;
+ }
+
+ /** The custom {@apilink StatisticsOptions.stateExtension|`stateExtension`} fields as they go into the record. */
+ #serializeStateExtension(extension: StateExtension): PersistedStateExtension {
+ const { serialize } = this.#stateExtension;
+
+ if (serialize === undefined) {
+ return extension as unknown as PersistedStateExtension;
+ }
+
+ // Unlike the way back, a failure here throws: the value is the caller's own, and a record written from a
+ // state that does not match its own declaration is not worth having.
+ return convertStateSync(serialize, extension, this.#persistStateKey);
+ }
+
+ /** Separates the declared custom fields from the built-in ones, so that each half goes through its own conversion. */
+ #splitState(state: StatisticState & StateExtension): { builtIn: StatisticState; extension: StateExtension } {
+ const builtIn: StatisticState & Partial<StateExtension> = { ...state };
+ const extension = {} as StateExtension;
+
+ for (const key of this.#stateExtensionKeys) {
+ extension[key] = state[key];
+ delete builtIn[key];
+ }
+
+ return { builtIn, extension };
}
/**
* Make this class serializable when called with `JSON.stringify(statsInstance)` directly
* or through `keyValueStore.setValue('KEY', statsInstance)`
*/
- toJSON(): StatisticPersistedState {
- return this.serializeState(this.state);
+ toJSON(): StatisticPersistedState & PersistedStateExtension {
+ return this.#serializeState(this.state);
}
}
/**
* Configuration for the {@apilink Statistics} instance used by the crawler
*/
-export interface StatisticsOptions {
+export interface StatisticsOptions<
+ StateExtension extends object = {},
+ PersistedStateExtension extends object = StateExtension,
+> {
/**
* Interval in seconds to log the current statistics
* @default 60
@@ -643,6 +801,64 @@ export interface StatisticsOptions {
* if crawler creation order changes.
*/
id?: string;
+
+ /**
+ * Custom fields to track alongside the built-in {@apilink StatisticState} ones. They become part of
+ * {@apilink Statistics.state|`state`} (typed as such), are persisted with the rest of the state, and are
+ * restored on migration or resurrect.
+ *
+ * ```ts
+ * const statistics = new Statistics({ stateExtension: { defaultState: { productsFound: 0 } } });
+ * statistics.state.productsFound++;
+ * ```
+ */
+ stateExtension?: StatisticStateExtensionOptions<StateExtension, PersistedStateExtension>;
+}
+
+/**
+ * How the custom fields of {@apilink StatisticsOptions.stateExtension} are initialized and converted to and from the
+ * persisted record - the same three things {@apilink RecoverableStateOptions} asks for, scoped to the custom half
+ * of the statistics state.
+ */
+export interface StatisticStateExtensionOptions<
+ StateExtension extends object,
+ PersistedStateExtension extends object = StateExtension,
+> {
+ /**
+ * The values the fields start with, and the ones {@apilink Statistics.reset} restores. A plain value is
+ * deep-copied with `structuredClone` each time it is used; pass a factory for a state `structuredClone` cannot
+ * rebuild.
+ *
+ * Can be omitted when `deserialize` supplies its own defaults, which is then the single place the fields are
+ * declared - see the example on {@apilink StatisticStateExtensionOptions.deserialize|`deserialize`}.
+ */
+ defaultState?: StateExtension | (() => StateExtension);
+
+ /**
+ * Rebuilds the custom fields from the persisted record, and the place to validate them before trusting them.
+ * Receives the whole record, so it has to supply a value for every field - `.default()` in a schema, or
+ * {@apilink StatisticStateExtensionOptions.defaultState|`defaultState`} alongside a conversion that copes with
+ * a missing field itself.
+ *
+ * ```ts
+ * const statistics = new Statistics({
+ * stateExtension: { deserialize: z.object({ productsFound: z.number().default(0) }) },
+ * });
+ * ```
+ *
+ * Without it, the declared fields are restored as they were persisted - which is a record off the key-value
+ * store taken at its word, `productsFound` included in whatever type it happens to hold.
+ *
+ * A conversion that rejects the record costs the custom fields their persisted values (they start from the
+ * defaults, with a warning) and nothing else.
+ */
+ deserialize?: SyncStateConversion<unknown, StateExtension>;
+
+ /**
+ * Converts the custom fields to the JSON-serializable form they are persisted in. Not needed for fields that
+ * already are one - pair it with `deserialize` for the fields that are not.
+ */
+ serialize?: SyncStateConversion<StateExtension, PersistedStateExtension>;
}
/**
diff --git a/packages/core/src/recoverable_state.ts b/packages/core/src/recoverable_state.ts
index b6a3cffaf..de8211191 100644
--- a/packages/core/src/recoverable_state.ts
+++ b/packages/core/src/recoverable_state.ts
@@ -16,6 +16,45 @@ const DEFAULT_PERSISTENCE_TIMEOUT_MILLIS = 60_000;
*/
export type StateConversion<TFrom, TTo> = ((value: TFrom) => Awaitable<TTo>) | StandardSchemaV1<TFrom, TTo>;
+/**
+ * A {@apilink StateConversion} for a caller that cannot await one - {@apilink Statistics}, whose `toJSON()` is
+ * synchronous, being the reason this exists.
+ *
+ * Only the function arm can be narrowed here: a Standard Schema is free to validate asynchronously, so a schema
+ * that does is rejected when it runs rather than when it is passed.
+ */
+export type SyncStateConversion<TFrom, TTo> = ((value: TFrom) => TTo) | StandardSchemaV1<TFrom, TTo>;
+
+/**
+ * Applies a {@apilink SyncStateConversion}, throwing a {@apilink StateValidationError} for a schema that rejects
+ * the value.
+ *
+ * @internal
+ */
+export function convertStateSync<TFrom, TTo>(
+ conversion: SyncStateConversion<TFrom, TTo>,
+ value: TFrom,
+ persistStateKey: string,
+): TTo {
+ if (typeof conversion === 'function') {
+ return conversion(value);
+ }
+
+ const result = conversion['~standard'].validate(value);
+
+ if ('then' in result) {
+ throw new Error(
+ `The state conversion for '${persistStateKey}' validated asynchronously, which this caller cannot await.`,
+ );
+ }
+
+ if (result.issues) {
+ throw new StateValidationError(persistStateKey, result.issues);
+ }
+
+ return result.value;
+}
+
export interface RecoverableStatePersistenceOptions {
/**
* The key under which the state is stored in the KeyValueStore
diff --git a/packages/http-crawler/src/internals/http-crawler.ts b/packages/http-crawler/src/internals/http-crawler.ts
index b5ae62a6c..b177fe797 100644
--- a/packages/http-crawler/src/internals/http-crawler.ts
+++ b/packages/http-crawler/src/internals/http-crawler.ts
@@ -84,7 +84,8 @@ export interface HttpCrawlerOptions<
ContextExtension = Dictionary<never>,
ExtendedContext extends Context = Context & ContextExtension,
Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>,
-> extends BasicCrawlerOptions<Context, ContextExtension, ExtendedContext, Routes> {
+ StatisticStateExtension extends object = {},
+> extends BasicCrawlerOptions<Context, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
/**
* Timeout for the whole navigation phase, given in seconds. A single window shared by the
* `preNavigationHooks`, the navigation (the HTTP request to the resource), and the `postNavigationHooks` -
@@ -350,7 +351,8 @@ export class HttpCrawler<
ContextExtension = Dictionary<never>,
ExtendedContext extends Context = Context & ContextExtension,
Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>,
-> extends BasicCrawler<Context, ContextExtension, ExtendedContext, Routes> {
+ StatisticStateExtension extends object = {},
+> extends BasicCrawler<Context, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
// Internal storage uses the base (non-extended) context types. The public option types are
// extension-aware for consumer DX, but internally the pipeline composes hooks against the
// concrete crawling context, which does not statically carry `ContextExtension`. The members
@@ -384,7 +386,7 @@ export class HttpCrawler<
* All `HttpCrawlerOptions` parameters are passed via an options object.
*/
constructor(
- options: HttpCrawlerOptions<Context, ContextExtension, ExtendedContext> &
+ options: HttpCrawlerOptions<Context, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> &
RequireContextPipeline<InternalHttpCrawlingContext, Context> = {} as any,
) {
const {
diff --git a/packages/jsdom-crawler/src/internals/jsdom-crawler.ts b/packages/jsdom-crawler/src/internals/jsdom-crawler.ts
index f67f1a595..77677e789 100644
--- a/packages/jsdom-crawler/src/internals/jsdom-crawler.ts
+++ b/packages/jsdom-crawler/src/internals/jsdom-crawler.ts
@@ -44,7 +44,14 @@ export interface JSDOMCrawlerOptions<
UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
Routes extends Record<keyof Routes, Dictionary> = Record<string, UserData>,
-> extends HttpCrawlerOptions<JSDOMCrawlingContext<UserData, JSONData>, ContextExtension, ExtendedContext, Routes> {
+ StatisticStateExtension extends object = {},
+> extends HttpCrawlerOptions<
+ JSDOMCrawlingContext<UserData, JSONData>,
+ ContextExtension,
+ ExtendedContext,
+ Routes,
+ StatisticStateExtension
+> {
/**
* Download and run scripts.
*/
@@ -203,7 +210,8 @@ export class JSDOMCrawler<
string,
GetUserDataFromRequest<JSDOMCrawlingContext['request']>
>,
-> extends HttpCrawler<JSDOMCrawlingContext, ContextExtension, ExtendedContext, Routes> {
+ StatisticStateExtension extends object = {},
+> extends HttpCrawler<JSDOMCrawlingContext, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
protected static override optionsShape = {
...HttpCrawler.optionsShape,
runScripts: z.boolean().optional(),
@@ -216,7 +224,9 @@ export class JSDOMCrawler<
#hideInternalConsole: boolean;
#virtualConsole: VirtualConsole | null = null;
- constructor(options: JSDOMCrawlerOptions<ContextExtension, ExtendedContext, any, any, Routes> = {}) {
+ constructor(
+ options: JSDOMCrawlerOptions<ContextExtension, ExtendedContext, any, any, Routes, StatisticStateExtension> = {},
+ ) {
const {
runScripts = false,
hideInternalConsole = false,
diff --git a/packages/linkedom-crawler/src/internals/linkedom-crawler.ts b/packages/linkedom-crawler/src/internals/linkedom-crawler.ts
index d9be04493..06ba73d3a 100644
--- a/packages/linkedom-crawler/src/internals/linkedom-crawler.ts
+++ b/packages/linkedom-crawler/src/internals/linkedom-crawler.ts
@@ -40,7 +40,14 @@ export interface LinkeDOMCrawlerOptions<
UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
Routes extends Record<keyof Routes, Dictionary> = Record<string, UserData>,
-> extends HttpCrawlerOptions<LinkeDOMCrawlingContext<UserData, JSONData>, ContextExtension, ExtendedContext, Routes> {}
+ StatisticStateExtension extends object = {},
+> extends HttpCrawlerOptions<
+ LinkeDOMCrawlingContext<UserData, JSONData>,
+ ContextExtension,
+ ExtendedContext,
+ Routes,
+ StatisticStateExtension
+> {}
export type LinkeDOMHook<
UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
@@ -184,10 +191,20 @@ export class LinkeDOMCrawler<
string,
GetUserDataFromRequest<LinkeDOMCrawlingContext['request']>
>,
-> extends HttpCrawler<LinkeDOMCrawlingContext, ContextExtension, ExtendedContext, Routes> {
+ StatisticStateExtension extends object = {},
+> extends HttpCrawler<LinkeDOMCrawlingContext, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
static #parser = new DOMParser();
- constructor(options: LinkeDOMCrawlerOptions<ContextExtension, ExtendedContext, any, any, Routes> = {}) {
+ constructor(
+ options: LinkeDOMCrawlerOptions<
+ ContextExtension,
+ ExtendedContext,
+ any,
+ any,
+ Routes,
+ StatisticStateExtension
+ > = {},
+ ) {
const { contextPipelineBuilder, ...rest } = options;
super({
diff --git a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts
index 14f013d6f..58dda9a6d 100644
--- a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts
+++ b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts
@@ -22,8 +22,7 @@ import type {
GetUserDataFromRequest,
RestrictedCrawlingContext,
RouterRoutes,
- StatisticPersistedState,
- StatisticState,
+ StatisticStateExtensionOptions,
StorageTransaction,
StorageTransactionView,
StorageWritePolicy,
@@ -59,58 +58,32 @@ type Result<TResult> =
| { result: TResult; ok: true; logs?: LogProxyCall[] }
| { error: unknown; ok: false; logs?: LogProxyCall[] };
-interface AdaptivePlaywrightCrawlerStatisticState extends StatisticState {
- httpOnlyRequestHandlerRuns?: number;
- browserRequestHandlerRuns?: number;
- renderingTypeMispredictions?: number;
-}
-
-interface AdaptivePlaywrightCrawlerPersistedStatisticState extends StatisticPersistedState {
- httpOnlyRequestHandlerRuns?: number;
- browserRequestHandlerRuns?: number;
- renderingTypeMispredictions?: number;
-}
-
-class AdaptivePlaywrightCrawlerStatistics extends Statistics {
- override get state(): AdaptivePlaywrightCrawlerStatisticState {
- return super.state;
- }
-
- protected override defaultState(): AdaptivePlaywrightCrawlerStatisticState {
- return {
- ...super.defaultState(),
- httpOnlyRequestHandlerRuns: 0,
- browserRequestHandlerRuns: 0,
- renderingTypeMispredictions: 0,
- };
- }
+const adaptiveStatisticStateSchema = z.object({
+ /** How many requests were handled by the HTTP-only request handler. */
+ httpOnlyRequestHandlerRuns: z.number().default(0),
- protected override deserializeState(
- persistedState: AdaptivePlaywrightCrawlerPersistedStatisticState,
- ): AdaptivePlaywrightCrawlerStatisticState {
- return {
- ...super.deserializeState(persistedState),
- httpOnlyRequestHandlerRuns: persistedState.httpOnlyRequestHandlerRuns,
- browserRequestHandlerRuns: persistedState.browserRequestHandlerRuns,
- renderingTypeMispredictions: persistedState.renderingTypeMispredictions,
- };
- }
+ /** How many requests were handled in a browser. */
+ browserRequestHandlerRuns: z.number().default(0),
- trackHttpOnlyRequestHandlerRun(): void {
- this.state.httpOnlyRequestHandlerRuns ??= 0;
- this.state.httpOnlyRequestHandlerRuns += 1;
- }
+ /** How many times the HTTP-only handler produced a result the `resultChecker` rejected. */
+ renderingTypeMispredictions: z.number().default(0),
+});
- trackBrowserRequestHandlerRun(): void {
- this.state.browserRequestHandlerRuns ??= 0;
- this.state.browserRequestHandlerRuns += 1;
- }
+/**
+ * The extra statistics fields {@apilink AdaptivePlaywrightCrawler} tracks on top of the built-in
+ * {@apilink StatisticState} ones. They are available on `crawler.stats.state` and are persisted with the rest of
+ * the statistics.
+ */
+export type AdaptivePlaywrightCrawlerStatisticState = z.infer<typeof adaptiveStatisticStateSchema>;
- trackRenderingTypeMisprediction(): void {
- this.state.renderingTypeMispredictions ??= 0;
- this.state.renderingTypeMispredictions += 1;
- }
-}
+/**
+ * The {@apilink AdaptivePlaywrightCrawlerStatisticState} fields as a {@apilink Statistics} state extension, defaults
+ * and all. A {@apilink Statistics} instance to be injected into an {@apilink AdaptivePlaywrightCrawler} has to carry
+ * them - `deserialize.extend()` your own fields onto this one and pass the result as `stateExtension`.
+ */
+export const adaptivePlaywrightCrawlerStatisticState = {
+ deserialize: adaptiveStatisticStateSchema,
+};
export interface AdaptivePlaywrightCrawlerContext<
UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
@@ -190,8 +163,15 @@ export interface AdaptivePlaywrightCrawlerOptions<
string,
GetUserDataFromRequest<AdaptivePlaywrightCrawlerContext['request']>
>,
+ StatisticStateExtension extends AdaptivePlaywrightCrawlerStatisticState = AdaptivePlaywrightCrawlerStatisticState,
> extends Omit<
- BasicCrawlerOptions<AdaptivePlaywrightCrawlerContext, ContextExtension, ExtendedContext, Routes>,
+ BasicCrawlerOptions<
+ AdaptivePlaywrightCrawlerContext,
+ ContextExtension,
+ ExtendedContext,
+ Routes,
+ StatisticStateExtension
+ >,
'preNavigationHooks' | 'postNavigationHooks'
> {
/**
@@ -313,7 +293,14 @@ export class AdaptivePlaywrightCrawler<
string,
GetUserDataFromRequest<AdaptivePlaywrightCrawlerContext['request']>
>,
-> extends BasicCrawler<AdaptivePlaywrightCrawlerContext, ContextExtension, ExtendedContext, Routes> {
+ StatisticStateExtension extends AdaptivePlaywrightCrawlerStatisticState = AdaptivePlaywrightCrawlerStatisticState,
+> extends BasicCrawler<
+ AdaptivePlaywrightCrawlerContext,
+ ContextExtension,
+ ExtendedContext,
+ Routes,
+ StatisticStateExtension
+> {
#renderingTypePredictor: OwnedOrInjected<IRenderingTypePredictor, RenderingTypePredictor>;
#resultChecker: NonNullable<AdaptivePlaywrightCrawlerOptions['resultChecker']>;
#shouldPropagateError: NonNullable<AdaptivePlaywrightCrawlerOptions['shouldPropagateError']>;
@@ -322,11 +309,6 @@ export class AdaptivePlaywrightCrawler<
#browserContextPipeline: ContextPipeline<CrawlingContext, ExtendedContext>;
#individualRequestHandlerTimeoutMillis: number;
- // The constructor always injects an `AdaptivePlaywrightCrawlerStatistics`, so narrowing the cast is sound.
- override get stats(): AdaptivePlaywrightCrawlerStatistics {
- return super.stats as AdaptivePlaywrightCrawlerStatistics;
- }
-
/**
* The write policy of the per-attempt transactions. Defaults the request queue to `deferred`:
* a discarded attempt's enqueues must never reach the queue.
@@ -335,7 +317,14 @@ export class AdaptivePlaywrightCrawler<
#teardownHooks: (() => Promise<unknown>)[] = [];
- constructor(options: AdaptivePlaywrightCrawlerOptions<ContextExtension, ExtendedContext, Routes> = {}) {
+ constructor(
+ options: AdaptivePlaywrightCrawlerOptions<
+ ContextExtension,
+ ExtendedContext,
+ Routes,
+ StatisticStateExtension
+ > = {},
+ ) {
const {
requestHandler,
renderingTypeDetectionRatio = 0.1,
@@ -363,6 +352,21 @@ export class AdaptivePlaywrightCrawler<
'AdaptivePlaywrightCrawlerOptions',
);
+ // The extra fields are only tracked if the injected instance was built with them - the types enforce that,
+ // but plain JS callers would otherwise silently increment `undefined` into a sticky `NaN`. Extend
+ // `adaptivePlaywrightCrawlerStatisticState` to satisfy this.
+ if (statistics !== undefined) {
+ parseArgument(
+ statistics.state,
+ z.object({
+ httpOnlyRequestHandlerRuns: z.number(),
+ browserRequestHandlerRuns: z.number(),
+ renderingTypeMispredictions: z.number(),
+ }),
+ 'statistics.state',
+ );
+ }
+
// Per-attempt buffering is load-bearing here: the handler runs up to twice per request and the
// losing attempt's writes must be discardable.
if (transactionalStorage === false) {
@@ -374,24 +378,21 @@ export class AdaptivePlaywrightCrawler<
);
}
- if (statistics !== undefined && !(statistics instanceof AdaptivePlaywrightCrawlerStatistics)) {
- throw new Error(
- 'AdaptivePlaywrightCrawler tracks extra fields on its own Statistics subclass and cannot use a ' +
- 'plain `statistics` instance. Omit the option to let the crawler build its own.',
- );
- }
-
super({
...rest,
errorHandler,
failedRequestHandler,
requestHandler,
requestHandlerTimeoutSecs,
- // Inject our subclass so the base tracks the extra adaptive fields instead of building a plain `Statistics`.
+ // The base would build a `Statistics` without the adaptive fields, so provide a default that has them.
+ // The cast covers a `StatisticStateExtension` that adds further fields - those can only come from an
+ // injected instance, in which case this default is never built.
statistics:
statistics ??
- new AdaptivePlaywrightCrawlerStatistics({
+ new Statistics({
logMessage: `${AdaptivePlaywrightCrawler.name} request statistics:`,
+ stateExtension:
+ adaptivePlaywrightCrawlerStatisticState as StatisticStateExtensionOptions<StatisticStateExtension>,
}),
contextPipelineBuilder: contextPipelineBuilder ?? (() => this.buildContextPipeline()),
// The base crawler must not wrap requests in a transaction of its own - this crawler opens
@@ -657,7 +658,7 @@ export class AdaptivePlaywrightCrawler<
try {
if (renderingTypePrediction.renderingType === 'static' && !shouldDetectRenderingType) {
crawlingContext.log.debug(`Running HTTP-only request handler for ${crawlingContext.request.url}`);
- this.stats.trackHttpOnlyRequestHandlerRun();
+ this.stats.state.httpOnlyRequestHandlerRuns++;
const plainHTTPRun = await this.crawlOne(
'static',
@@ -692,12 +693,12 @@ export class AdaptivePlaywrightCrawler<
crawlingContext.log.warning(
`HTTP-only request handler returned a suspicious result for ${crawlingContext.request.url}`,
);
- this.stats.trackRenderingTypeMisprediction();
+ this.stats.state.renderingTypeMispredictions++;
}
}
crawlingContext.log.debug(`Running browser request handler for ${crawlingContext.request.url}`);
- this.stats.trackBrowserRequestHandlerRun();
+ this.stats.state.browserRequestHandlerRuns++;
// Run the request handler in a browser. The copy of the crawler state is kept so that we can perform
// a rendering type detection if necessary. Without this measure, the HTTP request handler would run
diff --git a/packages/playwright-crawler/src/internals/playwright-crawler.ts b/packages/playwright-crawler/src/internals/playwright-crawler.ts
index d7b83be57..eaf11d867 100644
--- a/packages/playwright-crawler/src/internals/playwright-crawler.ts
+++ b/packages/playwright-crawler/src/internals/playwright-crawler.ts
@@ -46,6 +46,7 @@ export interface PlaywrightCrawlerOptions<
string,
GetUserDataFromRequest<PlaywrightCrawlingContext['request']>
>,
+ StatisticStateExtension extends object = {},
> extends BrowserCrawlerOptions<
Page,
Response,
@@ -53,7 +54,8 @@ export interface PlaywrightCrawlerOptions<
ContextExtension,
ExtendedContext,
{ browserPlugins: [PlaywrightPlugin] },
- Routes
+ Routes,
+ StatisticStateExtension
> {
/**
* The same options as used by {@apilink launchPlaywright}.
@@ -199,6 +201,7 @@ export class PlaywrightCrawler<
string,
GetUserDataFromRequest<PlaywrightCrawlingContext['request']>
>,
+ StatisticStateExtension extends object = {},
> extends BrowserCrawler<
Page,
Response,
@@ -207,7 +210,8 @@ export class PlaywrightCrawler<
PlaywrightCrawlingContext,
ContextExtension,
ExtendedContext,
- Routes
+ Routes,
+ StatisticStateExtension
> {
protected static override optionsShape = {
...BrowserCrawler.optionsShape,
@@ -220,7 +224,9 @@ export class PlaywrightCrawler<
/**
* All `PlaywrightCrawler` parameters are passed via an options object.
*/
- constructor(options: PlaywrightCrawlerOptions<ContextExtension, ExtendedContext, Routes> = {}) {
+ constructor(
+ options: PlaywrightCrawlerOptions<ContextExtension, ExtendedContext, Routes, StatisticStateExtension> = {},
+ ) {
const parsedOptions = parseArgument(options, PlaywrightCrawler.optionsSchema, 'PlaywrightCrawlerOptions');
const { launchContext, headless, contextPipelineBuilder, ...browserCrawlerOptions } = parsedOptions;
@@ -252,7 +258,12 @@ export class PlaywrightCrawler<
browserPoolOptions.browserPlugins = [playwrightLauncher.createBrowserPlugin()];
super({
- ...(browserCrawlerOptions as unknown as PlaywrightCrawlerOptions<ContextExtension, ExtendedContext>),
+ ...(browserCrawlerOptions as unknown as PlaywrightCrawlerOptions<
+ ContextExtension,
+ ExtendedContext,
+ Routes,
+ StatisticStateExtension
+ >),
launchContext,
browserPoolOptions,
contextPipelineBuilder: contextPipelineBuilder ?? (() => this.buildContextPipeline()),
diff --git a/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts b/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts
index fe8f3ff73..16fb43bdd 100644
--- a/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts
+++ b/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts
@@ -47,6 +47,7 @@ export interface PuppeteerCrawlerOptions<
string,
GetUserDataFromRequest<PuppeteerCrawlingContext['request']>
>,
+ StatisticStateExtension extends object = {},
> extends BrowserCrawlerOptions<
Page,
HTTPResponse,
@@ -54,7 +55,8 @@ export interface PuppeteerCrawlerOptions<
ContextExtension,
ExtendedContext,
{ browserPlugins: [PuppeteerPlugin] },
- Routes
+ Routes,
+ StatisticStateExtension
> {
/**
* Options used by {@apilink launchPuppeteer} to start new Puppeteer instances.
@@ -176,6 +178,7 @@ export class PuppeteerCrawler<
string,
GetUserDataFromRequest<PuppeteerCrawlingContext['request']>
>,
+ StatisticStateExtension extends object = {},
> extends BrowserCrawler<
Page,
HTTPResponse,
@@ -184,7 +187,8 @@ export class PuppeteerCrawler<
PuppeteerCrawlingContext,
ContextExtension,
ExtendedContext,
- Routes
+ Routes,
+ StatisticStateExtension
> {
protected static override optionsShape = {
...BrowserCrawler.optionsShape,
@@ -196,7 +200,9 @@ export class PuppeteerCrawler<
/**
* All `PuppeteerCrawler` parameters are passed via an options object.
*/
- constructor(options: PuppeteerCrawlerOptions<ContextExtension, ExtendedContext, Routes> = {}) {
+ constructor(
+ options: PuppeteerCrawlerOptions<ContextExtension, ExtendedContext, Routes, StatisticStateExtension> = {},
+ ) {
const parsedOptions = parseArgument(options, PuppeteerCrawler.optionsSchema, 'PuppeteerCrawlerOptions');
const { launchContext, headless, proxyConfiguration, contextPipelineBuilder, ...browserCrawlerOptions } =
@@ -234,7 +240,10 @@ export class PuppeteerCrawler<
HTTPResponse,
PuppeteerCrawlingContext,
ContextExtension,
- ExtendedContext
+ ExtendedContext,
+ BrowserPoolOptions,
+ Routes,
+ StatisticStateExtension
>),
launchContext,
proxyConfiguration,
diff --git a/packages/stagehand-crawler/src/internals/stagehand-crawler.ts b/packages/stagehand-crawler/src/internals/stagehand-crawler.ts
index 560898f09..5a26e61a1 100644
--- a/packages/stagehand-crawler/src/internals/stagehand-crawler.ts
+++ b/packages/stagehand-crawler/src/internals/stagehand-crawler.ts
@@ -252,6 +252,7 @@ export interface StagehandCrawlerOptions<
string,
GetUserDataFromRequest<StagehandCrawlingContext['request']>
>,
+ StatisticStateExtension extends object = {},
> extends BrowserCrawlerOptions<
StagehandPage,
Response,
@@ -259,7 +260,8 @@ export interface StagehandCrawlerOptions<
ContextExtension,
ExtendedContext,
{ browserPlugins: [StagehandPlugin] },
- Routes
+ Routes,
+ StatisticStateExtension
> {
/**
* Stagehand-specific configuration options.
@@ -395,6 +397,7 @@ export class StagehandCrawler<
string,
GetUserDataFromRequest<StagehandCrawlingContext['request']>
>,
+ StatisticStateExtension extends object = {},
> extends BrowserCrawler<
StagehandPage,
Response,
@@ -403,7 +406,8 @@ export class StagehandCrawler<
StagehandCrawlingContext,
ContextExtension,
ExtendedContext,
- Routes
+ Routes,
+ StatisticStateExtension
> {
protected static override optionsShape = {
...BrowserCrawler.optionsShape,
@@ -418,7 +422,9 @@ export class StagehandCrawler<
*
* @param options - Crawler configuration options
*/
- constructor(options: StagehandCrawlerOptions<ContextExtension, ExtendedContext, Routes> = {}) {
+ constructor(
+ options: StagehandCrawlerOptions<ContextExtension, ExtendedContext, Routes, StatisticStateExtension> = {},
+ ) {
const parsedOptions = parseArgument(options, StagehandCrawler.optionsSchema, 'StagehandCrawlerOptions');
const {
@@ -442,7 +448,12 @@ export class StagehandCrawler<
// Initialize BrowserCrawler with Stagehand plugin and fingerprinting enabled
super({
- ...(browserCrawlerOptions as StagehandCrawlerOptions<ContextExtension, ExtendedContext>),
+ ...(browserCrawlerOptions as StagehandCrawlerOptions<
+ ContextExtension,
+ ExtendedContext,
+ Routes,
+ StatisticStateExtension
+ >),
launchContext,
browserPoolOptions,
contextPipelineBuilder: contextPipelineBuilder ?? (() => this.buildContextPipeline()),
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index d4796eca7..40e4b6d79 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -501,6 +501,9 @@ importers:
'@sapphire/async-queue':
specifier: ^1.5.5
version: 1.5.5
+ '@standard-schema/spec':
+ specifier: ^1.0.0
+ version: 1.1.0
'@vladfrangu/async_event_emitter':
specifier: ^2.4.6
version: 2.4.7
diff --git a/test/core/crawlers/adaptive_playwright_crawler.test.ts b/test/core/crawlers/adaptive_playwright_crawler.test.ts
index 562a97271..a43035895 100644
--- a/test/core/crawlers/adaptive_playwright_crawler.test.ts
+++ b/test/core/crawlers/adaptive_playwright_crawler.test.ts
@@ -10,6 +10,7 @@ import {
KeyValueStore,
MemoryStorageBackend,
serviceLocator,
+ Statistics,
} from '@crawlee/core';
import type {
AdaptivePlaywrightCrawlerContext,
@@ -19,6 +20,7 @@ import type {
} from '@crawlee/playwright';
import {
AdaptivePlaywrightCrawler,
+ adaptivePlaywrightCrawlerStatisticState,
BasicCrawler,
createAdaptivePlaywrightRouter,
fullResultComparator,
@@ -442,6 +444,47 @@ describe('AdaptivePlaywrightCrawler', () => {
expect(requestHandler).toHaveBeenCalledTimes(2);
expect(resultChecker).toHaveBeenCalledTimes(1);
+
+ expect(crawler.stats.state).toMatchObject({
+ httpOnlyRequestHandlerRuns: 1,
+ renderingTypeMispredictions: 1,
+ browserRequestHandlerRuns: 1,
+ });
+ });
+
+ test('should track its own fields alongside those of an injected statistics instance', async () => {
+ const statistics = new Statistics({
+ stateExtension: {
+ deserialize: adaptivePlaywrightCrawlerStatisticState.deserialize.extend({
+ productsFound: z.number().default(0),
+ }),
+ },
+ });
+
+ // Instantiated directly (rather than via `makeOneshotCrawler`) so that the `StatisticStateExtension`
+ // generic is inferred from the injected instance rather than defaulted by the helper's parameter type.
+ const crawler = new AdaptivePlaywrightCrawler({
+ statistics,
+ renderingTypeDetectionRatio: 0.1,
+ maxConcurrency: 1,
+ maxRequestRetries: 0,
+ maxRequestsPerCrawl: 1,
+ requestList: await RequestList.open({
+ sources: [new URL(`http://${HOSTNAME}:${port}/static`).toString()],
+ }),
+ renderingTypePredictor: makeRiggedRenderingTypePredictor({
+ detectionProbabilityRecommendation: 0,
+ renderingType: 'static',
+ }),
+ requestHandler: async ({ querySelectorAll }) => {
+ statistics.state.productsFound += (await querySelectorAll('h1')).length;
+ },
+ });
+
+ await crawler.run();
+
+ expect(crawler.stats.state.productsFound).toEqual(1);
+ expect(crawler.stats.state.httpOnlyRequestHandlerRuns).toEqual(1);
});
describe('shouldPropagateError', () => {
diff --git a/test/core/crawlers/basic_crawler.test.ts b/test/core/crawlers/basic_crawler.test.ts
index 11f03ca85..ac672be5e 100644
--- a/test/core/crawlers/basic_crawler.test.ts
+++ b/test/core/crawlers/basic_crawler.test.ts
@@ -2178,6 +2178,27 @@ describe('BasicCrawler', () => {
expect(customStats.state.requestsFinished).toBe(1);
});
+ it('exposes the custom state fields of a supplied instance on crawler.stats', async () => {
+ const stats = new Statistics({
+ persistenceOptions: { enable: false },
+ stateExtension: { defaultState: { productsFound: 0 } },
+ });
+
+ const crawler = new BasicCrawler({
+ statistics: stats,
+ requestHandler: async () => {
+ stats.state.productsFound += 2;
+ },
+ });
+
+ await crawler.run([{ url: 'https://example.com' }]);
+
+ // typed through the crawler, not just through the instance we constructed
+ expect(crawler.stats.state.productsFound).toBe(2);
+ // @ts-expect-error only the fields declared in `defaultState` are exposed
+ void crawler.stats.state.categoriesFound;
+ });
+
it('does not reset a supplied instance between runs, but does reset a default', async () => {
const stats = new Statistics({ persistenceOptions: { enable: false } });
const injectingCrawler = new BasicCrawler({
diff --git a/test/core/crawlers/statistics.test.ts b/test/core/crawlers/statistics.test.ts
index ec10960ed..7709ce7b2 100644
--- a/test/core/crawlers/statistics.test.ts
+++ b/test/core/crawlers/statistics.test.ts
@@ -1,6 +1,8 @@
-import type { StatisticPersistedState, StatisticState } from '@crawlee/core';
+import type { StatisticsOptions } from '@crawlee/core';
+import type { StatisticPersistedState } from '@crawlee/core';
import { EventType, KeyValueStore, MemoryStorageBackend, serviceLocator, Statistics } from '@crawlee/core';
import type { Dictionary } from '@crawlee/types';
+import { z } from 'zod';
describe('Statistics', () => {
const getPerMinute = (jobCount: number, totalTickMillis: number) => {
@@ -36,12 +38,16 @@ describe('Statistics', () => {
describe('persist state', () => {
// needs to go first for predictability
- test('should increment id by each new consecutive instance', () => {
+ test('should increment id by each new consecutive instance', async () => {
expect(stats.id).toEqual('0');
// @ts-expect-error Accessing private prop
expect(Statistics.id).toEqual(1);
- // @ts-expect-error Accessing private prop
- expect(stats.persistStateKey).toEqual('CRAWLEE_CRAWLER_STATISTICS_0');
+
+ // the id is what the record is keyed by
+ await stats.startCapturing();
+ await stats.stopCapturing();
+ expect(await store.getValue('CRAWLEE_CRAWLER_STATISTICS_0')).not.toBeNull();
+
const [n1, n2] = [new Statistics(), new Statistics()];
expect(n1.id).toEqual('1');
expect(n2.id).toEqual('2');
@@ -150,30 +156,6 @@ describe('Statistics', () => {
await stats.stopCapturing();
});
- test('should persist the extra fields of a subclass', async () => {
- type ExtendedState = StatisticState & { productsFound: number };
-
- class ExtendedStatistics extends Statistics {
- override get state(): ExtendedState {
- return super.state as ExtendedState;
- }
-
- protected override defaultState(): ExtendedState {
- return { ...super.defaultState(), productsFound: 0 };
- }
- }
-
- const extended = new ExtendedStatistics({ id: 'extended' });
- await extended.startCapturing();
- extended.state.productsFound = 7;
- await extended.persistState();
-
- // The record schema knows nothing about them, and must not drop them on the way through.
- expect(await store.getValue(persistStateKey(extended))).toMatchObject({ productsFound: 7 });
-
- await extended.stopCapturing();
- });
-
test('should keep the shape of the persisted record', async () => {
stats.startJob(0);
vitest.advanceTimersByTime(100);
@@ -448,6 +430,174 @@ describe('Statistics', () => {
await stats.stopCapturing();
});
+ describe('custom state fields', () => {
+ /** Persists `productsFound: 7` and one finished request under `id`. */
+ const persistCustomState = async (id: string) => {
+ const stats = new Statistics({ id, stateExtension: { defaultState: { productsFound: 0 } } });
+
+ await stats.startCapturing();
+ stats.startJob(0);
+ stats.finishJob(0, 0);
+ stats.state.productsFound = 7;
+ await stats.stopCapturing();
+ };
+
+ /** Loads whatever was persisted under `options.id` and returns the resulting state. */
+ const loadState = async <T extends object, P extends object = T>(options: StatisticsOptions<T, P>) => {
+ const stats = new Statistics(options);
+
+ await stats.startCapturing();
+ await stats.stopCapturing();
+
+ return stats.state;
+ };
+
+ test('should expose the custom fields and restore their defaults on reset', () => {
+ const stats = new Statistics({
+ stateExtension: { defaultState: { productsFound: 0, seenPerDomain: {} as Record<string, number> } },
+ });
+
+ stats.state.productsFound += 3;
+ stats.state.seenPerDomain['example.com'] = 1;
+ stats.reset();
+
+ expect(stats.state.productsFound).toEqual(0);
+ // the nested default is cloned per reset, not shared with the previous state
+ expect(stats.state.seenPerDomain).toEqual({});
+
+ // @ts-expect-error `categoriesFound` was not declared
+ stats.state.categoriesFound = 1;
+ });
+
+ test('should restore the persisted custom fields, keeping the defaults of ones the record lacks', async () => {
+ await persistCustomState('grown-stats');
+
+ const state = await loadState({
+ id: 'grown-stats',
+ // `categoriesFound` was declared after the state above was persisted
+ stateExtension: { defaultState: { productsFound: 0, categoriesFound: 42 } },
+ });
+
+ expect(state.productsFound).toEqual(7);
+ expect(state.categoriesFound).toEqual(42);
+ });
+
+ test('should reject a custom field that collides with a built-in one', () => {
+ expect(() => new Statistics({ stateExtension: { defaultState: { requestsFinished: 999 } } })).toThrow(
+ /`requestsFinished` collides with a built-in one/,
+ );
+ });
+
+ test('should keep a default value that JSON cannot carry', () => {
+ const stats = new Statistics({ stateExtension: { defaultState: { budget: Infinity } } });
+
+ stats.state.budget -= 1;
+ stats.reset();
+
+ // `structuredClone`, as in RecoverableState - the JSON the record is written as would make this a `null`
+ expect(stats.state.budget).toEqual(Infinity);
+ });
+
+ describe('with a deserialize conversion', () => {
+ const productsFound = z.object({ productsFound: z.number().default(0) });
+
+ test('should derive the defaults from the conversion', () => {
+ const stats = new Statistics({ stateExtension: { deserialize: productsFound } });
+
+ expect(stats.state.productsFound).toEqual(0);
+
+ stats.state.productsFound += 3;
+ stats.reset();
+
+ expect(stats.state.productsFound).toEqual(0);
+ });
+
+ test('should throw when the defaults cannot be derived', () => {
+ expect(
+ () => new Statistics({ stateExtension: { deserialize: z.object({ productsFound: z.number() }) } }),
+ ).toThrow(/Could not derive the default values .* give every field a default/s);
+ });
+
+ test('should restore the custom fields it validates', async () => {
+ await persistCustomState('validated-stats');
+
+ const state = await loadState({
+ id: 'validated-stats',
+ stateExtension: { deserialize: productsFound },
+ });
+
+ expect(state.productsFound).toEqual(7);
+ });
+
+ test('should start only the custom fields from scratch on a record it rejects', async () => {
+ await persistCustomState('corrupt-custom-stats');
+
+ const record = (await store.getValue<Dictionary>(`CRAWLEE_CRAWLER_STATISTICS_corrupt-custom-stats`))!;
+ await store.setValue(`CRAWLEE_CRAWLER_STATISTICS_corrupt-custom-stats`, {
+ ...record,
+ productsFound: 'plenty',
+ });
+
+ const stats = new Statistics({
+ id: 'corrupt-custom-stats',
+ stateExtension: { deserialize: productsFound },
+ });
+ // @ts-expect-error Accessing private prop
+ const warningSpy = vitest.spyOn(stats.log, 'warning').mockImplementation(() => {});
+
+ await stats.startCapturing();
+
+ expect(warningSpy).toHaveBeenCalledWith(
+ expect.stringContaining('starting those from scratch'),
+ expect.objectContaining({ persistStateKey: 'CRAWLEE_CRAWLER_STATISTICS_corrupt-custom-stats' }),
+ );
+ // the custom field lost its value, the crawler's own counters did not
+ expect(stats.state.productsFound).toEqual(0);
+ expect(stats.state.requestsFinished).toEqual(1);
+
+ await stats.stopCapturing();
+ });
+
+ test('should take the record at its word without one', async () => {
+ await persistCustomState('trusted-stats');
+
+ const record = (await store.getValue<Dictionary>(`CRAWLEE_CRAWLER_STATISTICS_trusted-stats`))!;
+ await store.setValue(`CRAWLEE_CRAWLER_STATISTICS_trusted-stats`, {
+ ...record,
+ productsFound: 'plenty',
+ });
+
+ const state = await loadState({
+ id: 'trusted-stats',
+ stateExtension: { defaultState: { productsFound: 0 } },
+ });
+
+ // the documented cost of declaring the fields without a conversion to check them with
+ expect(state.productsFound).toEqual('plenty');
+ });
+ });
+
+ test('should round-trip a field that is not JSON through a serialize/deserialize pair', async () => {
+ const stateExtension = {
+ deserialize: z.object({ lastSeenAt: z.coerce.date().default(() => new Date(0)) }),
+ serialize: ({ lastSeenAt }: { lastSeenAt: Date }) => ({ lastSeenAt: lastSeenAt.toISOString() }),
+ };
+
+ const stats = new Statistics({ id: 'date-stats', stateExtension });
+ await stats.startCapturing();
+ stats.state.lastSeenAt = new Date('2020-01-01T00:00:00.000Z');
+ await stats.stopCapturing();
+
+ expect(await store.getValue<Dictionary>('CRAWLEE_CRAWLER_STATISTICS_date-stats')).toMatchObject({
+ lastSeenAt: '2020-01-01T00:00:00.000Z',
+ });
+
+ const restored = await loadState({ id: 'date-stats', stateExtension });
+
+ expect(restored.lastSeenAt).toEqual(new Date('2020-01-01T00:00:00.000Z'));
+ });
+ });
+
describe('explicit id option', () => {
test('statistics with same explicit id should share persisted state', async () => {
const stats1 = new Statistics({ id: 'shared-stats' });
diff --git a/website/sidebars.js b/website/sidebars.js
index f0f6ac01c..ebdb30468 100644
--- a/website/sidebars.js
+++ b/website/sidebars.js
@@ -51,6 +51,7 @@ module.exports = {
'guides/stagehand-crawler-guide',
'guides/running-in-web-server/running-in-web-server',
'guides/parallel-scraping/parallel-scraping-guide',
+ 'guides/custom-statistics',
'guides/custom-http-client/custom-http-client',
'guides/custom-logger/custom-logger'
],