Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

refactor!: Rename BasicCrawler.stats to statistics (#4028)

Jan Buchar41a2584d434a
diff --git a/docs/guides/custom_statistics.mdx b/docs/guides/custom_statistics.mdx
index 49227ec85..4e00dbcfd 100644
--- a/docs/guides/custom_statistics.mdx
+++ b/docs/guides/custom_statistics.mdx
@@ -11,9 +11,9 @@ import BasicSource from '!!raw-loader!./custom_statistics_basic.ts';
import ValidatedSource from '!!raw-loader!./custom_statistics_validated.ts';
import AdaptiveSource from '!!raw-loader!./custom_statistics_adaptive.ts';
-Every crawler collects run statistics — requests finished, failures, retries, durations — into a <ApiLink to="core/class/Statistics">`Statistics`</ApiLink> instance exposed as `crawler.stats`. You can track your own counters in the same place, and get the persistence, logging and lifecycle handling that the built-in fields already have.
+Every crawler collects run statistics — requests finished, failures, retries, durations — into a <ApiLink to="core/class/Statistics">`Statistics`</ApiLink> instance exposed as `crawler.statistics`. You can track your own counters in the same place, and get the persistence, logging and lifecycle handling that the built-in fields already have.
-Keeping a counter in a plain variable works right up until the run is interrupted. Statistics state is written to the key-value store and restored on migration or resurrect, so a counter that lives in `stats.state` survives a run that a plain variable would not.
+Keeping a counter in a plain variable works right up until the run is interrupted. Statistics state is written to the key-value store and restored on migration or resurrect, so a counter that lives in `statistics.state` survives a run that a plain variable would not.
## Declaring custom fields
@@ -21,7 +21,7 @@ Pass <ApiLink to="core/interface/StatisticsOptions#stateExtension">`stateExtensi
<CodeBlock className="language-typescript">{BasicSource}</CodeBlock>
-The field types flow through to `crawler.stats.state`, so they are available anywhere you hold the crawler, not just where you hold the statistics instance. Reading an undeclared field is a compile-time error.
+The field types flow through to `crawler.statistics.state`, so they are available anywhere you hold the crawler, not just where you hold the statistics instance. Reading an undeclared field is a compile-time error.
<ApiLink to="core/interface/StatisticStateExtensionOptions#defaultState">`defaultState`</ApiLink> is also what `reset()` restores, and it determines which keys are read back from a persisted record. A field you add later simply keeps its default when an older record is restored, so growing the state between runs does not need a migration.
diff --git a/docs/guides/custom_statistics_adaptive.ts b/docs/guides/custom_statistics_adaptive.ts
index 69b0c29e7..6596e3f08 100644
--- a/docs/guides/custom_statistics_adaptive.ts
+++ b/docs/guides/custom_statistics_adaptive.ts
@@ -19,4 +19,4 @@ const crawler = new AdaptivePlaywrightCrawler({
await crawler.run(['https://crawlee.dev']);
-console.log(`Handled ${crawler.stats.state.httpOnlyRequestHandlerRuns} requests without a browser`);
+console.log(`Handled ${crawler.statistics.state.httpOnlyRequestHandlerRuns} requests without a browser`);
diff --git a/docs/guides/custom_statistics_basic.ts b/docs/guides/custom_statistics_basic.ts
index 643b9a32f..2342c794b 100644
--- a/docs/guides/custom_statistics_basic.ts
+++ b/docs/guides/custom_statistics_basic.ts
@@ -16,5 +16,5 @@ const crawler = new CheerioCrawler({
await crawler.run(['https://crawlee.dev']);
-// The custom fields are typed on `crawler.stats` too.
-console.log(`Found ${crawler.stats.state.productsFound} products`);
+// The custom fields are typed on `crawler.statistics` too.
+console.log(`Found ${crawler.statistics.state.productsFound} products`);
diff --git a/docs/public-api/crawlee-basic.api.md b/docs/public-api/crawlee-basic.api.md
index 3e65c021d..9ddff7730 100644
--- a/docs/public-api/crawlee-basic.api.md
+++ b/docs/public-api/crawlee-basic.api.md
@@ -219,7 +219,7 @@ export class BasicCrawler<Context extends CrawlingContext = CrawlingContext, Con
protected runRequestHandler(crawlingContext: ExtendedContext): Promise<void>;
get sessionPool(): ISessionPool;
setStatusMessage(message: string, options?: SetStatusMessageOptions): void;
- get stats(): IStatistics<StatisticStateExtension>;
+ get statistics(): IStatistics<StatisticStateExtension>;
stop(reason?: string): void;
teardown(): Promise<void>;
protected throwOnBlockedRequest(statusCode: number): void;
diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.md
index c839f0045..8b02f15dc 100644
--- a/docs/upgrading/upgrading_v4.md
+++ b/docs/upgrading/upgrading_v4.md
@@ -53,6 +53,7 @@ The purely mechanical renames, collected in one place. Where a row links to a se
| `enqueueLinks({ globs, regexps, pseudoUrls })` | `enqueueLinks({ include })` ([details](#globs-regexps-and-pseudourls-replaced-by-include)) |
| `(await enqueueLinks()).processedRequests` | `(await enqueueLinks()).addedRequests` ([details](#enqueuelinks-return-value-reshaped-addrequestsbatchedresult-instead-of-batchaddrequestsresult)) |
| `autoscaledPoolOptions` | `taskLoopOptions` ([narrowed](#autoscaledpooloptions-is-now-taskloopoptions-and-no-longer-carries-concurrency-config)) |
+| `crawler.stats` | `crawler.statistics` ([retyped](#statisticsoptions-is-replaced-by-a-statistics-instance)) |
| `browserPoolOptions` | `browserPool` + a `*BrowserPool()` factory ([details](#browserpooloptions-is-removed)) |
| `gotScraping` (from `@crawlee/utils`) | `GotScrapingHttpClient` (`@crawlee/got-scraping-client`) |
| `SDK_`-prefixed internal KVS keys | `CRAWLEE_`-prefixed ([details](#internal-kvs-keys-renamed)) |
@@ -1260,7 +1261,7 @@ const crawler = new BasicCrawler({
Omit the option and the crawler builds its own default, exactly as before. A supplied instance is treated as borrowed: the crawler records into it and drives its capture lifecycle for the run, but never `reset()`s it between `run()` calls — so a preconfigured instance keeps whatever state it was handed.
-The option accepts the built-in `Statistics` or any object implementing the new `IStatistics` interface, so a fully custom statistics backend can be plugged in without subclassing. The crawler exposes it as `crawler.stats` typed as `IStatistics`.
+The option accepts the built-in `Statistics` or any object implementing the new `IStatistics` interface, so a fully custom statistics backend can be plugged in without subclassing. The crawler exposes it as `crawler.statistics` (renamed from `crawler.stats`) typed as `IStatistics`.
### The `Statistics` persistence lifecycle is stricter
@@ -1274,7 +1275,7 @@ A persisted record is also validated on load now. One that does not match the ex
A consequence of the hooks going away: the persisted record is now validated strictly, and keys that are neither built-in nor declared in `stateExtension` are dropped rather than written back. `calculate()` is still public and still an override point.
-The custom field types reach `crawler.stats.state` through a new trailing `StatisticStateExtension` type parameter on the crawler classes and their options. It defaults to `{}`, so existing type arguments keep working — except on `BrowserCrawler` and `BrowserCrawlerOptions`, where it was inserted after `Routes` and shifts the trailing internal parameters (`GoToOptions`, `__BrowserPlugins`, …). Adjust any explicit type arguments you passed to those two.
+The custom field types reach `crawler.statistics.state` through a new trailing `StatisticStateExtension` type parameter on the crawler classes and their options. It defaults to `{}`, so existing type arguments keep working — except on `BrowserCrawler` and `BrowserCrawlerOptions`, where it was inserted after `Routes` and shifts the trailing internal parameters (`GoToOptions`, `__BrowserPlugins`, …). Adjust any explicit type arguments you passed to those two.
`AdaptivePlaywrightCrawler` now uses this mechanism for its own extra fields, with two consequences:
@@ -2035,7 +2036,7 @@ The full list of removed exports and members, for ctrl-F purposes. Where a repla
- `HttpCrawler._parseHTML` (protected)
- `HttpCrawler.use` and the `CrawlerExtension` class (experimental) - the `ContextPipeline` should be used for extending the crawler
- `BasicCrawler._tagUserHandlerError` (protected) - internal error-tagging helper, no longer part of the crawler surface
-- `BasicCrawler.handledRequestsCount` setter (`@deprecated`) - the throw-on-assign guard is gone; the getter is now internal-only and the count is derived from `this.stats`
+- `BasicCrawler.handledRequestsCount` setter (`@deprecated`) - the throw-on-assign guard is gone; the getter is now internal-only and the count is derived from `this.statistics`
- `PlaywrightPlugin._containerProxyServer` (public) - was an unused, never-populated field
- `Snapshotter._snapshotMemory`, `Snapshotter._memoryOverloadWarning`, `Snapshotter._snapshotEventLoop`, `Snapshotter._snapshotCpu`, `Snapshotter._snapshotClient`, `Snapshotter._pruneSnapshots` (all `@deprecated` protected stubs) - snapshotting is now handled by the individual load signals, and the `Snapshotter` itself is internal to `ConcurrencySystem`; there is no longer a public API for reading raw resource snapshots
- `FileDownloadOptions.streamHandler` - streaming should now be handled directly in the `requestHandler` instead
diff --git a/packages/basic-crawler/src/internals/basic-crawler.ts b/packages/basic-crawler/src/internals/basic-crawler.ts
index 1151b0f37..4111db951 100644
--- a/packages/basic-crawler/src/internals/basic-crawler.ts
+++ b/packages/basic-crawler/src/internals/basic-crawler.ts
@@ -510,7 +510,7 @@ export interface BasicCrawlerOptions<
* implementing {@apilink IStatistics}.
*
* Custom fields declared via {@apilink StatisticsOptions.stateExtension|`stateExtension`} are carried over to
- * {@apilink BasicCrawler.stats|`crawler.stats.state`}:
+ * {@apilink BasicCrawler.statistics|`crawler.statistics.state`}:
*
* ```ts
* const statistics = new Statistics({ stateExtension: { defaultState: { productsFound: 0 } } });
@@ -523,8 +523,8 @@ export interface BasicCrawlerOptions<
* });
*
* await crawler.run();
- * // the custom fields are typed on `crawler.stats` too
- * console.log(crawler.stats.state.productsFound);
+ * // the custom fields are typed on `crawler.statistics` too
+ * console.log(crawler.statistics.state.productsFound);
* ```
*/
statistics?: IStatistics<StatisticStateExtension>;
@@ -715,15 +715,15 @@ export class BasicCrawler<
*/
static #useStateAnonymousIndices = new Set<number>();
- /** Backs the {@apilink BasicCrawler.stats|`stats`} getter. */
- #statsDep: OwnedOrInjected<IStatistics<StatisticStateExtension>, Statistics<StatisticStateExtension>>;
+ /** Backs the {@apilink BasicCrawler.statistics|`statistics`} getter. */
+ #statisticsDep: OwnedOrInjected<IStatistics<StatisticStateExtension>, Statistics<StatisticStateExtension>>;
/**
* The statistics instance collecting the crawler's run statistics - either the injected `statistics` option or a
* crawler-built default. Typed as {@apilink IStatistics} so custom implementations can be plugged in.
*/
- get stats(): IStatistics<StatisticStateExtension> {
- return this.#statsDep.value;
+ get statistics(): IStatistics<StatisticStateExtension> {
+ return this.#statisticsDep.value;
}
/**
@@ -858,7 +858,7 @@ export class BasicCrawler<
protected readonly maxRequestsPerCrawl?: number;
private get handledRequestsCount(): number {
- return this.stats.state.requestsFinished + this.stats.state.requestsFailed;
+ return this.statistics.state.requestsFinished + this.statistics.state.requestsFailed;
}
#statusMessageLoggingInterval: number;
@@ -1130,7 +1130,7 @@ export class BasicCrawler<
this.maxRequestRetries = maxRequestRetries;
this.maxCrawlDepth = maxCrawlDepth;
this.#sameDomainDelaySecs = sameDomainDelaySecs;
- this.#statsDep = OwnedOrInjected.resolve<
+ this.#statisticsDep = OwnedOrInjected.resolve<
IStatistics<StatisticStateExtension>,
Statistics<StatisticStateExtension>
>(
@@ -1208,7 +1208,7 @@ export class BasicCrawler<
// Started here, rather than in `handleRequest`, so that a failure during context pipeline
// initialization (e.g. a browser page timing out before the request handler ever runs) is
// still accounted for by `failJob` below - which is a no-op without a matching `startJob`.
- this.stats.startJob(request.id || request.uniqueKey);
+ this.statistics.startJob(request.id || request.uniqueKey);
const crawlingContext = { request } as { request: Request } & Partial<CrawlingContext>;
try {
@@ -1230,7 +1230,7 @@ export class BasicCrawler<
// ContextPipelineInterruptedError means the request was intentionally skipped
// (e.g., doesn't match enqueue strategy after redirect). Just return gracefully.
if (error instanceof ContextPipelineInterruptedError) {
- this.stats.discardJob(request.id || request.uniqueKey);
+ this.statistics.discardJob(request.id || request.uniqueKey);
await this.timeoutAndRetry(
async () => this.requestManager?.markRequestAsHandled(request),
this.internalTimeoutMillis,
@@ -1582,13 +1582,13 @@ export class BasicCrawler<
}
private getPeriodicLogger() {
- let previousState = { ...this.stats.state };
+ let previousState = { ...this.statistics.state };
const getOperationMode = (): { mode: 'ERROR' | 'REGULAR'; failedDelta: number } => {
- const { requestsFailed } = this.stats.state;
+ const { requestsFailed } = this.statistics.state;
const { requestsFailed: previousRequestsFailed } = previousState;
- previousState = { ...this.stats.state };
+ previousState = { ...this.statistics.state };
const failedDelta = requestsFailed - previousRequestsFailed;
@@ -1607,15 +1607,15 @@ export class BasicCrawler<
message = `Experiencing problems, ${failedDelta} failed requests in the past ${this.#statusMessageLoggingInterval} seconds.`;
} else {
const total = await this.requestManager?.getTotalCount();
- message = `Crawled ${this.stats.state.requestsFinished}${total ? `/${total}` : ''} pages, ${
- this.stats.state.requestsFailed
+ message = `Crawled ${this.statistics.state.requestsFinished}${total ? `/${total}` : ''} pages, ${
+ this.statistics.state.requestsFailed
} failed requests, desired concurrency ${this.concurrencySystem?.desiredConcurrency ?? 0}.`;
}
if (this.#statusMessageCallback) {
await this.#statusMessageCallback({
crawler: this as any,
- state: this.stats.state,
+ state: this.statistics.state,
previousState,
message,
});
@@ -1673,7 +1673,7 @@ export class BasicCrawler<
}
// A supplied statistics instance keeps whatever state it was handed - only wipe a default we built.
- await this.#statsDep.ifOwned(async (stats) => {
+ await this.#statisticsDep.ifOwned(async (stats) => {
stats.reset();
await stats.resetStore();
});
@@ -1696,7 +1696,7 @@ export class BasicCrawler<
try {
await this.init();
- await this.stats.startCapturing();
+ await this.statistics.startCapturing();
} catch (error) {
// Clean up here before propagating, otherwise a failed startup would leave the process hanging.
await this.teardown().catch((teardownError) => {
@@ -1731,30 +1731,30 @@ export class BasicCrawler<
try {
await this.#autoscaledPool!.run();
} finally {
- await this.stats.stopCapturing();
+ await this.statistics.stopCapturing();
await this.teardown();
process.off('SIGINT', sigintHandler);
eventManager.off(EventType.MIGRATING, boundPauseOnMigration);
eventManager.off(EventType.ABORTING, boundPauseOnMigration);
- const finalStats = this.stats.calculate();
+ const finalStats = this.statistics.calculate();
stats = {
- requestsFinished: this.stats.state.requestsFinished,
- requestsFailed: this.stats.state.requestsFailed,
- retryHistogram: this.stats.requestRetryHistogram,
+ requestsFinished: this.statistics.state.requestsFinished,
+ requestsFailed: this.statistics.state.requestsFailed,
+ retryHistogram: this.statistics.requestRetryHistogram,
...finalStats,
};
this.log.info('Final request statistics:', stats as unknown as Record<string, unknown>);
- if (this.stats.errorTracker.total !== 0) {
+ if (this.statistics.errorTracker.total !== 0) {
const prettify = ([count, info]: [number, string[]]) =>
`${count}x: ${info.at(-1)!.trim()} (${info[0]})`;
this.log.info(`Error analysis:`, {
- totalErrors: this.stats.errorTracker.total,
- uniqueErrors: this.stats.errorTracker.getUniqueErrorCount(),
- mostCommonErrors: this.stats.errorTracker.getMostPopularErrors(3).map(prettify),
+ totalErrors: this.statistics.errorTracker.total,
+ uniqueErrors: this.statistics.errorTracker.getUniqueErrorCount(),
+ mostCommonErrors: this.statistics.errorTracker.getMostPopularErrors(3).map(prettify),
});
}
@@ -1773,9 +1773,9 @@ export class BasicCrawler<
periodicLogger.stop();
this.setStatusMessage(
- `Finished! Total ${this.stats.state.requestsFinished + this.stats.state.requestsFailed} requests: ${
- this.stats.state.requestsFinished
- } succeeded, ${this.stats.state.requestsFailed} failed.`,
+ `Finished! Total ${this.statistics.state.requestsFinished + this.statistics.state.requestsFailed} requests: ${
+ this.statistics.state.requestsFinished
+ } succeeded, ${this.statistics.state.requestsFailed} failed.`,
{ isStatusMessageTerminal: true, level: 'INFO' },
);
@@ -2539,7 +2539,7 @@ export class BasicCrawler<
}
})();
- await Promise.all([requestManagerPersistPromise, this.stats.persistState?.()]);
+ await Promise.all([requestManagerPersistPromise, this.statistics.persistState?.()]);
}
/**
@@ -2586,7 +2586,7 @@ export class BasicCrawler<
);
isRequestLocked = false; // markRequestAsHandled succeeded and unlocked the request
- this.stats.finishJob(statisticsId, request.retryCount);
+ this.statistics.finishJob(statisticsId, request.retryCount);
// reclaim session if request finishes successfully
request.state = RequestState.DONE;
@@ -2752,7 +2752,7 @@ export class BasicCrawler<
const shouldRetryRequest = this.canRequestBeRetried(request, error);
if (shouldRetryRequest) {
- await this.stats.errorTrackerRetry.addAsync(error, crawlingContext);
+ await this.statistics.errorTrackerRetry.addAsync(error, crawlingContext);
await this.errorHandler?.(
crawlingContext as CrawlingContext & Partial<ExtendedContext>, // valid cast - ExtendedContext transitively extends CrawlingContext
error,
@@ -2790,16 +2790,16 @@ export class BasicCrawler<
// This is to make sure the error snapshot is not duplicated in the errorTrackerRetry and errorTracker objects.
const { noRetry, maxRetries } = request;
if (noRetry || !maxRetries) {
- await this.stats.errorTracker.addAsync(error, crawlingContext);
+ await this.statistics.errorTracker.addAsync(error, crawlingContext);
} else {
- this.stats.errorTracker.add(error);
+ this.statistics.errorTracker.add(error);
}
// If we get here, the request is either not retryable
// or failed more than retryCount times and will not be retried anymore.
// Mark the request as failed and do not retry.
await source.markRequestAsHandled(request);
- this.stats.failJob(request.id || request.uniqueKey, request.retryCount);
+ this.statistics.failJob(request.id || request.uniqueKey, request.retryCount);
await this.handleFailedRequestHandler(crawlingContext, error); // This function prints an error message.
}
diff --git a/packages/browser-crawler/src/internals/browser-crawler.ts b/packages/browser-crawler/src/internals/browser-crawler.ts
index fa8d46042..553ce7df6 100644
--- a/packages/browser-crawler/src/internals/browser-crawler.ts
+++ b/packages/browser-crawler/src/internals/browser-crawler.ts
@@ -829,7 +829,7 @@ export abstract class BrowserCrawler<
if (typeof response === 'object' && typeof response.status === 'function') {
const status: number = response.status();
- this.stats.registerStatusCode(status);
+ this.statistics.registerStatusCode(status);
// Ahead of the error-status throw below: a 429 the user opted into treating as an error is still a
// rate limit the domain should back off from.
diff --git a/packages/http-crawler/src/internals/http-crawler.ts b/packages/http-crawler/src/internals/http-crawler.ts
index b177fe797..02872fe6a 100644
--- a/packages/http-crawler/src/internals/http-crawler.ts
+++ b/packages/http-crawler/src/internals/http-crawler.ts
@@ -723,7 +723,7 @@ export class HttpCrawler<
const contentType = { type, encoding };
if (status >= 400 && status <= 599) {
- this.stats.registerStatusCode(status);
+ this.statistics.registerStatusCode(status);
}
if (this.isErrorStatusCode(status)) {
diff --git a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts
index 58dda9a6d..f0d30a425 100644
--- a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts
+++ b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts
@@ -71,7 +71,7 @@ const adaptiveStatisticStateSchema = z.object({
/**
* The extra statistics fields {@apilink AdaptivePlaywrightCrawler} tracks on top of the built-in
- * {@apilink StatisticState} ones. They are available on `crawler.stats.state` and are persisted with the rest of
+ * {@apilink StatisticState} ones. They are available on `crawler.statistics.state` and are persisted with the rest of
* the statistics.
*/
export type AdaptivePlaywrightCrawlerStatisticState = z.infer<typeof adaptiveStatisticStateSchema>;
@@ -658,7 +658,7 @@ export class AdaptivePlaywrightCrawler<
try {
if (renderingTypePrediction.renderingType === 'static' && !shouldDetectRenderingType) {
crawlingContext.log.debug(`Running HTTP-only request handler for ${crawlingContext.request.url}`);
- this.stats.state.httpOnlyRequestHandlerRuns++;
+ this.statistics.state.httpOnlyRequestHandlerRuns++;
const plainHTTPRun = await this.crawlOne(
'static',
@@ -693,12 +693,12 @@ export class AdaptivePlaywrightCrawler<
crawlingContext.log.warning(
`HTTP-only request handler returned a suspicious result for ${crawlingContext.request.url}`,
);
- this.stats.state.renderingTypeMispredictions++;
+ this.statistics.state.renderingTypeMispredictions++;
}
}
crawlingContext.log.debug(`Running browser request handler for ${crawlingContext.request.url}`);
- this.stats.state.browserRequestHandlerRuns++;
+ this.statistics.state.browserRequestHandlerRuns++;
// Run the request handler in a browser. The copy of the crawler state is kept so that we can perform
// a rendering type detection if necessary. Without this measure, the HTTP request handler would run
diff --git a/test/core/crawlers/adaptive_playwright_crawler.test.ts b/test/core/crawlers/adaptive_playwright_crawler.test.ts
index a43035895..8d4ec7239 100644
--- a/test/core/crawlers/adaptive_playwright_crawler.test.ts
+++ b/test/core/crawlers/adaptive_playwright_crawler.test.ts
@@ -445,7 +445,7 @@ describe('AdaptivePlaywrightCrawler', () => {
expect(requestHandler).toHaveBeenCalledTimes(2);
expect(resultChecker).toHaveBeenCalledTimes(1);
- expect(crawler.stats.state).toMatchObject({
+ expect(crawler.statistics.state).toMatchObject({
httpOnlyRequestHandlerRuns: 1,
renderingTypeMispredictions: 1,
browserRequestHandlerRuns: 1,
@@ -483,8 +483,8 @@ describe('AdaptivePlaywrightCrawler', () => {
await crawler.run();
- expect(crawler.stats.state.productsFound).toEqual(1);
- expect(crawler.stats.state.httpOnlyRequestHandlerRuns).toEqual(1);
+ expect(crawler.statistics.state.productsFound).toEqual(1);
+ expect(crawler.statistics.state.httpOnlyRequestHandlerRuns).toEqual(1);
});
describe('shouldPropagateError', () => {
diff --git a/test/core/crawlers/basic_crawler.test.ts b/test/core/crawlers/basic_crawler.test.ts
index abf816d67..a40b2b1ce 100644
--- a/test/core/crawlers/basic_crawler.test.ts
+++ b/test/core/crawlers/basic_crawler.test.ts
@@ -816,7 +816,7 @@ describe('BasicCrawler', () => {
});
// need to monkeypatch the stats class, otherwise it will never finish
- basicCrawler.stats.persistState = async () => Promise.resolve();
+ basicCrawler.statistics.persistState = async () => Promise.resolve();
await persistPromise;
expect(finished).toBe(false);
@@ -2129,7 +2129,7 @@ describe('BasicCrawler', () => {
requestHandler: async () => {},
});
- expect(crawler.stats).toBeInstanceOf(Statistics);
+ expect(crawler.statistics).toBeInstanceOf(Statistics);
});
it('records into a supplied Statistics instance', async () => {
@@ -2140,7 +2140,7 @@ describe('BasicCrawler', () => {
requestHandler: async () => {},
});
- expect(crawler.stats).toBe(stats);
+ expect(crawler.statistics).toBe(stats);
await crawler.run();
@@ -2170,7 +2170,7 @@ describe('BasicCrawler', () => {
};
const crawler = new BasicCrawler({ statistics: customStats, requestHandler: async () => {} });
- expect(crawler.stats).toBe(customStats);
+ expect(crawler.statistics).toBe(customStats);
await crawler.run([{ url: 'https://example.com' }]);
@@ -2178,7 +2178,7 @@ describe('BasicCrawler', () => {
expect(customStats.state.requestsFinished).toBe(1);
});
- it('exposes the custom state fields of a supplied instance on crawler.stats', async () => {
+ it('exposes the custom state fields of a supplied instance on crawler.statistics', async () => {
const stats = new Statistics({
persistenceOptions: { enable: false },
stateExtension: { defaultState: { productsFound: 0 } },
@@ -2194,9 +2194,9 @@ describe('BasicCrawler', () => {
await crawler.run([{ url: 'https://example.com' }]);
// typed through the crawler, not just through the instance we constructed
- expect(crawler.stats.state.productsFound).toBe(2);
+ expect(crawler.statistics.state.productsFound).toBe(2);
// @ts-expect-error only the fields declared in `defaultState` are exposed
- void crawler.stats.state.categoriesFound;
+ void crawler.statistics.state.categoriesFound;
});
it('does not reset a supplied instance between runs, but does reset a default', async () => {
@@ -2217,16 +2217,16 @@ describe('BasicCrawler', () => {
const owningCrawler = new BasicCrawler({
requestHandler: async () => {},
});
- // `crawler.stats` is typed as `IStatistics`, which omits the owned-only `reset()` - the default is a
+ // `crawler.statistics` is typed as `IStatistics`, which omits the owned-only `reset()` - the default is a
// concrete `Statistics`, so cast to spy on it.
- const ownedResetSpy = vitest.spyOn(owningCrawler.stats as Statistics, 'reset');
+ const ownedResetSpy = vitest.spyOn(owningCrawler.statistics as Statistics, 'reset');
await owningCrawler.run([{ url: 'https://example.com', uniqueKey: 'owned-1' }]);
await owningCrawler.run([{ url: 'https://example.com', uniqueKey: 'owned-2' }]);
// A crawler-owned default is wiped at the start of each run.
expect(ownedResetSpy).toHaveBeenCalled();
- expect(owningCrawler.stats.state.requestsFinished).toBe(1);
+ expect(owningCrawler.statistics.state.requestsFinished).toBe(1);
});
});
@@ -2404,7 +2404,7 @@ describe('BasicCrawler', () => {
requestHandler: async () => {},
});
- crawler.stats.state.requestsFinished = 2;
+ crawler.statistics.state.requestsFinished = 2;
// Try to add 6 requests - should only add 3 due to limit
const requestsToAdd = [
@@ -2437,7 +2437,7 @@ describe('BasicCrawler', () => {
requestHandler: async () => {},
});
- crawler.stats.state.requestsFinished = 1;
+ crawler.statistics.state.requestsFinished = 1;
// First call - should add 2 requests (2 more slots to go)
await crawler.addRequests(['http://example.com/1', 'http://example.com/2']);
@@ -2486,7 +2486,7 @@ describe('BasicCrawler', () => {
requestHandler: async () => {},
});
- crawler.stats.state.requestsFinished = 0;
+ crawler.statistics.state.requestsFinished = 0;
// Mock robots.txt checking to disallow some URLs
vitest.spyOn(crawler as any, 'isAllowedBasedOnRobotsTxtFile').mockImplementation(async (url) => {
@@ -2695,7 +2695,7 @@ describe('BasicCrawler', () => {
await crawler.run(['http://example.com/1']);
await crawler.run(['http://example.com/1']);
- expect(crawler.stats.state.requestsFinished).toBe(1);
+ expect(crawler.statistics.state.requestsFinished).toBe(1);
});
test('refuses to be combined with a request manager that throttles on its own', async () => {
@@ -2813,7 +2813,7 @@ describe('BasicCrawler', () => {
return;
}
- crawler.stats.state.requestsFinished = 2;
+ crawler.statistics.state.requestsFinished = 2;
await context.addRequests(requestsToAdd, { label: 'not-undefined' });
},
@@ -3500,8 +3500,8 @@ describe('BasicCrawler', () => {
await crawlerA.run([{ url: `http://${HOSTNAME}:${port}` }]);
await crawlerB.run([{ url: `http://${HOSTNAME}:${port}` }]);
- expect(crawlerA.stats.state.requestsFinished).toBe(1);
- expect(crawlerB.stats.state.requestsFinished).toBe(1);
+ expect(crawlerA.statistics.state.requestsFinished).toBe(1);
+ expect(crawlerB.statistics.state.requestsFinished).toBe(1);
});
});