Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

refactor!: Extract service management from Configuration into ServiceLocator class (#3325)

Jan Bucharbcde2b034872
diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.md
index d4d65a0f7..d26f0791e 100644
--- a/docs/upgrading/upgrading_v4.md
+++ b/docs/upgrading/upgrading_v4.md
@@ -40,6 +40,7 @@ The crawler following options are removed:
- `BasicCrawler._cleanupContext` (protected) - this is now handled by the `ContextPipeline`
- `BasicCrawler.isRequestBlocked` (protected)
+- `BasicCrawler.events` (protected) - this should be accessed via `BasicCrawler.serviceLocator`
- `BrowserRequestHandler` and `BrowserErrorHandler` types in `@crawlee/browser`
- `BrowserCrawler.userProvidedRequestHandler` (protected)
- `BrowserCrawler.requestHandlerTimeoutInnerMillis` (protected)
@@ -115,6 +116,97 @@ The `KeyValueStore.getPublicUrl` method is now asynchronous and reads the public
The `preNavigationHooks` option in `HttpCrawler` subclasses no longer accepts the `gotOptions` object as a second parameter. Modify the `crawlingContext` fields (e.g. `.request`) directly instead.
+## Service management moved from `Configuration` to `ServiceLocator`
+
+The service management functionality has been extracted from `Configuration` into a new `ServiceLocator` class, following the pattern established in Crawlee for Python.
+
+### Breaking changes
+
+The following methods and properties have been removed from `Configuration`:
+
+- `Configuration.getStorageClient()` - moved to `ServiceLocator.getStorageClient()`
+- `Configuration.getEventManager()` - moved to `ServiceLocator.getEventManager()`
+- `Configuration.useStorageClient()` - use `ServiceLocator.setStorageClient()` instead
+- `Configuration.useEventManager()` - use `ServiceLocator.setEventManager()` instead
+- `Configuration.storageManagers` - moved to `ServiceLocator.storageManagers`
+
+The `EventManager` and `LocalEventManager` constructors now accept an options object for configuring event intervals (e.g. `persistStateIntervalMillis`, `systemInfoIntervalMillis`). You can also use the new `LocalEventManager.fromConfig()` factory method to create an instance with intervals derived from a `Configuration` object.
+
+### Migration guide
+
+If you were using the removed `Configuration` methods directly, you need to update your code:
+
+**Before:**
+```typescript
+import { Configuration } from 'crawlee';
+
+const config = Configuration.getGlobalConfig();
+const storageClient = config.getStorageClient();
+const eventManager = config.getEventManager();
+
+// or static methods
+const storageClient = Configuration.getStorageClient();
+```
+
+**After:**
+```typescript
+import { serviceLocator } from 'crawlee';
+
+const storageClient = serviceLocator.getStorageClient();
+const eventManager = serviceLocator.getEventManager();
+```
+
+### Using per-crawler services (recommended)
+
+The new `ServiceLocator` supports per-crawler service isolation, allowing you to use different storage clients or event managers for different crawlers by passing them via options:
+
+```typescript
+import { BasicCrawler, Configuration, LocalEventManager } from 'crawlee';
+import { MemoryStorage } from '@crawlee/memory-storage';
+
+const crawler = new BasicCrawler({
+ requestHandler: async ({ request, log }) => {
+ log.info(`Processing ${request.url}`);
+ },
+ configuration: new Configuration({ headless: false }),
+ storageClient: new MemoryStorage(),
+ eventManager: LocalEventManager.fromConfig(),
+});
+
+await crawler.run(['https://example.com']);
+```
+
+### Using the global service locator
+
+For most use cases, the global `serviceLocator` singleton works well:
+
+```typescript
+import { serviceLocator, BasicCrawler } from 'crawlee';
+import { MemoryStorage } from '@crawlee/memory-storage';
+
+// Configure global services (optional)
+serviceLocator.setStorageClient(new MemoryStorage());
+
+// All crawlers will use the global service locator by default
+const crawler = new BasicCrawler({
+ requestHandler: async ({ request, log }) => {
+ log.info(`Processing ${request.url}`);
+ },
+});
+```
+
+### Accessing configuration
+
+`Configuration.getGlobalConfig()` remains as a utility function, but in most cases, you should use `serviceLocator.getConfiguration()` instead:
+
+```typescript
+import { serviceLocator } from 'crawlee';
+
+const config = serviceLocator.getConfiguration();
+```
+
+Do note that the method is currently misnamed - in specific circumstances, it will not return the global configuration object, but the one from the currently active service locator.
+
## `transformRequestFunction` precedence in `enqueueLinks`
The `transformRequestFunction` callback in `enqueueLinks` now runs **after** URL pattern filtering (`globs`, `regexps`, `pseudoUrls`) instead of before. This means it has the highest priority and can overwrite any request options set by patterns or the global `label` option.
diff --git a/packages/basic-crawler/src/internals/basic-crawler.ts b/packages/basic-crawler/src/internals/basic-crawler.ts
index be944e57b..b94c6dcdb 100644
--- a/packages/basic-crawler/src/internals/basic-crawler.ts
+++ b/packages/basic-crawler/src/internals/basic-crawler.ts
@@ -1,10 +1,15 @@
import { writeFile } from 'node:fs/promises';
import { dirname } from 'node:path';
-
+import { LruCache } from '@apify/datastructures';
+import type { Log } from '@apify/log';
+import defaultLog, { LogLevel } from '@apify/log';
+import { addTimeoutToPromise, TimeoutError, tryCancel } from '@apify/timeout';
+import { cryptoRandomObjectId } from '@apify/utilities';
import type {
AddRequestsBatchedOptions,
AddRequestsBatchedResult,
AutoscaledPoolOptions,
+ Configuration,
CrawlingContext,
DatasetExportOptions,
EnqueueLinksOptions,
@@ -23,21 +28,21 @@ import type {
SessionPoolOptions,
SkippedRequestCallback,
Source,
- StatisticsOptions,
StatisticState,
+ StatisticsOptions,
} from '@crawlee/core';
import {
AutoscaledPool,
- Configuration,
+ bindMethodsToServiceLocator,
ContextPipeline,
ContextPipelineCleanupError,
ContextPipelineInitializationError,
ContextPipelineInterruptedError,
CriticalError,
Dataset,
- enqueueLinks,
EnqueueStrategy,
EventType,
+ enqueueLinks,
KeyValueStore,
mergeCookies,
NonRetryableError,
@@ -51,9 +56,11 @@ import {
RequestState,
RetryRequestError,
Router,
+ ServiceLocator,
SessionError,
SessionPool,
Statistics,
+ serviceLocator,
validateUserData,
validators,
} from '@crawlee/core';
@@ -65,6 +72,7 @@ import type {
Dictionary,
ProxyInfo,
SetStatusMessageOptions,
+ StorageClient,
} from '@crawlee/types';
import { getObjectType, isAsyncIterable, isIterable, RobotsTxtFile, ROTATE_PROXY_ERRORS, sleep } from '@crawlee/utils';
import { stringify } from 'csv-stringify/sync';
@@ -73,12 +81,6 @@ import ow from 'ow';
import { getDomain } from 'tldts';
import type { ReadonlyDeep, SetRequired } from 'type-fest';
-import { LruCache } from '@apify/datastructures';
-import type { Log } from '@apify/log';
-import defaultLog, { LogLevel } from '@apify/log';
-import { addTimeoutToPromise, TimeoutError, tryCancel } from '@apify/timeout';
-import { cryptoRandomObjectId } from '@apify/utilities';
-
import { createSendRequest } from './send-request.js';
export interface BasicCrawlingContext<UserData extends Dictionary = Dictionary> extends CrawlingContext<UserData> {}
@@ -397,6 +399,24 @@ export interface BasicCrawlerOptions<
*/
proxyConfiguration?: ProxyConfiguration;
+ /**
+ * Custom configuration to use for this crawler.
+ * If provided, the crawler will use its own ServiceLocator instance instead of the global one.
+ */
+ configuration?: Configuration;
+
+ /**
+ * Custom storage client to use for this crawler.
+ * If provided, the crawler will use its own ServiceLocator instance instead of the global one.
+ */
+ storageClient?: StorageClient;
+
+ /**
+ * Custom event manager to use for this crawler.
+ * If provided, the crawler will use its own ServiceLocator instance instead of the global one.
+ */
+ eventManager?: EventManager;
+
/**
* A unique identifier for the crawler instance. This ID is used to isolate the state returned by
* {@apilink BasicCrawler.useState|`crawler.useState()`} from other crawler instances.
@@ -599,7 +619,6 @@ export class BasicCrawler<
protected sessionPoolOptions: SessionPoolOptions;
protected useSessionPool: boolean;
protected autoscaledPoolOptions: AutoscaledPoolOptions;
- protected events: EventManager;
protected httpClient: BaseHttpClient;
protected retryOnBlocked: boolean;
protected respectRobotsTxtFile: boolean | { userAgent?: string };
@@ -643,6 +662,10 @@ export class BasicCrawler<
onSkippedRequest: ow.optional.function,
httpClient: ow.optional.object,
+ configuration: ow.optional.object,
+ storageClient: ow.optional.object,
+ eventManager: ow.optional.object,
+
// AutoscaledPool shorthands
minConcurrency: ow.optional.number,
maxConcurrency: ow.optional.number,
@@ -664,7 +687,6 @@ export class BasicCrawler<
constructor(
options: BasicCrawlerOptions<Context, ContextExtension, ExtendedContext> &
RequireContextPipeline<CrawlingContext, Context> = {} as any, // cast because the constructor logic handles missing `contextPipelineBuilder` - the type is just for DX
- readonly config = Configuration.getGlobalConfig(),
) {
ow(options, 'BasicCrawlerOptions', ow.object.exactShape(BasicCrawler.optionsShape));
@@ -683,6 +705,11 @@ export class BasicCrawler<
useSessionPool = true,
proxyConfiguration,
+ // Service locator options
+ configuration,
+ storageClient,
+ eventManager,
+
// AutoscaledPool shorthands
minConcurrency,
maxConcurrency,
@@ -707,206 +734,226 @@ export class BasicCrawler<
id,
} = options;
- // Store whether the user explicitly provided an ID
- this.hasExplicitId = id !== undefined;
- // Store the user-provided ID, or generate a unique one for tracking purposes (not for state key)
- this.crawlerId = id ?? cryptoRandomObjectId();
+ // Create per-crawler service locator if custom services were provided.
+ // This wraps every method on the crawler instance so that calls to the global `serviceLocator`
+ // (via AsyncLocalStorage) resolve to this scoped instance instead.
+ // We also enter the scope for the rest of the constructor body, so that any code below
+ // that accesses `serviceLocator` will see the correct (scoped) instance.
+ let serviceLocatorScope = { enterScope: () => {}, exitScope: () => {} };
+
+ if (
+ storageClient ||
+ eventManager ||
+ (configuration !== undefined && configuration !== serviceLocator.getConfiguration())
+ ) {
+ const scopedServiceLocator = new ServiceLocator(configuration, eventManager, storageClient);
+ serviceLocatorScope = bindMethodsToServiceLocator(scopedServiceLocator, this);
+ }
- // Store the builder so that it can be run when the contextPipeline is needed.
- // Invoking it immediately would cause problems with parent constructor call order.
- this.contextPipelineBuilder = () => {
- let contextPipeline = (options.contextPipelineBuilder?.() ??
- ContextPipeline.create<CrawlingContext>()) as ContextPipeline<CrawlingContext, Context>; // Thanks to the RequireContextPipeline, contextPipeline will only be undefined if InitialContextType is CrawlingContext
+ try {
+ serviceLocatorScope.enterScope();
+
+ // Store whether the user explicitly provided an ID
+ this.hasExplicitId = id !== undefined;
+ // Store the user-provided ID, or generate a unique one for tracking purposes (not for state key)
+ this.crawlerId = id ?? cryptoRandomObjectId();
+
+ // Store the builder so that it can be run when the contextPipeline is needed.
+ // Invoking it immediately would cause problems with parent constructor call order.
+ this.contextPipelineBuilder = () => {
+ let contextPipeline = (options.contextPipelineBuilder?.() ??
+ ContextPipeline.create<CrawlingContext>()) as ContextPipeline<CrawlingContext, Context>; // Thanks to the RequireContextPipeline, contextPipeline will only be undefined if InitialContextType is CrawlingContext
+
+ if (options.extendContext !== undefined) {
+ contextPipeline = contextPipeline.compose({
+ action: async (context) => await options.extendContext(context),
+ });
+ }
- if (options.extendContext !== undefined) {
contextPipeline = contextPipeline.compose({
- action: async (context) => await options.extendContext(context),
+ action: async (context) => {
+ const { request } = context;
+ if (!this.requestMatchesEnqueueStrategy(request)) {
+ // eslint-disable-next-line dot-notation
+ const message = `Skipping request ${request.id} (starting url: ${request.url} -> loaded url: ${request.loadedUrl}) because it does not match the enqueue strategy (${request['enqueueStrategy']}).`;
+ this.log.debug(message);
+
+ request.noRetry = true;
+ request.state = RequestState.SKIPPED;
+
+ await this.handleSkippedRequest({ url: request.url, reason: 'redirect' });
+
+ throw new ContextPipelineInterruptedError(message);
+ }
+ return context;
+ },
});
- }
- contextPipeline = contextPipeline.compose({
- action: async (context) => {
- const { request } = context;
- if (!this.requestMatchesEnqueueStrategy(request)) {
- // eslint-disable-next-line dot-notation
- const message = `Skipping request ${request.id} (starting url: ${request.url} -> loaded url: ${request.loadedUrl}) because it does not match the enqueue strategy (${request['enqueueStrategy']}).`;
- this.log.debug(message);
-
- request.noRetry = true;
- request.state = RequestState.SKIPPED;
+ return contextPipeline as ContextPipeline<CrawlingContext, ExtendedContext>;
+ };
- await this.handleSkippedRequest({ url: request.url, reason: 'redirect' });
+ if (requestManager !== undefined) {
+ if (requestList !== undefined || requestQueue !== undefined) {
+ throw new Error(
+ 'The `requestManager` option cannot be used in conjunction with `requestList` and/or `requestQueue`',
+ );
+ }
+ this.requestManager = requestManager;
+ this.requestQueue = requestManager as RequestProvider; // TODO(v4) - the cast is not fully legitimate here, but it's fine for internal usage by the BasicCrawler
+ } else {
+ this.requestList = requestList;
+ this.requestQueue = requestQueue;
+ }
- throw new ContextPipelineInterruptedError(message);
- }
- return context;
- },
- });
+ this.httpClient = httpClient ?? new GotScrapingHttpClient();
+ this.proxyConfiguration = proxyConfiguration;
+ this.log = log;
+ this.statusMessageLoggingInterval = statusMessageLoggingInterval;
+ this.statusMessageCallback = statusMessageCallback as StatusMessageCallback;
+ this.domainAccessedTime = new Map();
+ this.experiments = experiments;
+ this.robotsTxtFileCache = new LruCache({ maxLength: 1000 });
+ this.handleSkippedRequest = this.handleSkippedRequest.bind(this);
+
+ this.requestHandler = requestHandler ?? this.router;
+ this.failedRequestHandler = failedRequestHandler;
+ this.errorHandler = errorHandler;
+
+ if (requestHandlerTimeoutSecs) {
+ this.requestHandlerTimeoutMillis = requestHandlerTimeoutSecs * 1000;
+ } else {
+ this.requestHandlerTimeoutMillis = 60_000;
+ }
- return contextPipeline as ContextPipeline<CrawlingContext, ExtendedContext>;
- };
+ this.retryOnBlocked = retryOnBlocked;
+ this.respectRobotsTxtFile = respectRobotsTxtFile;
+ this.onSkippedRequest = onSkippedRequest;
+
+ const tryEnv = (val?: string) => (val == null ? null : +val);
+ // allow at least 5min for internal timeouts
+ this.internalTimeoutMillis =
+ tryEnv(process.env.CRAWLEE_INTERNAL_TIMEOUT) ?? Math.max(this.requestHandlerTimeoutMillis * 2, 300e3);
+
+ // override the default internal timeout of request queue to respect `requestHandlerTimeoutMillis`
+ if (this.requestQueue) {
+ this.requestQueue.internalTimeoutMillis = this.internalTimeoutMillis;
+ // for request queue v2, we want to lock requests for slightly longer than the request handler timeout so that there is some padding for locking-related overhead,
+ // but never for less than a minute
+ this.requestQueue.requestLockSecs = Math.max(this.requestHandlerTimeoutMillis / 1000 + 5, 60);
+ }
- if (requestManager !== undefined) {
- if (requestList !== undefined || requestQueue !== undefined) {
- throw new Error(
- 'The `requestManager` option cannot be used in conjunction with `requestList` and/or `requestQueue`',
- );
+ this.maxRequestRetries = maxRequestRetries;
+ this.maxCrawlDepth = maxCrawlDepth;
+ this.sameDomainDelayMillis = sameDomainDelaySecs * 1000;
+ this.maxSessionRotations = maxSessionRotations;
+ this.stats = new Statistics({
+ logMessage: `${log.getOptions().prefix} request statistics:`,
+ log,
+ ...(this.hasExplicitId ? { id: this.crawlerId } : {}),
+ ...statisticsOptions,
+ });
+ this.sessionPoolOptions = {
+ ...sessionPoolOptions,
+ log,
+ };
+ if (this.retryOnBlocked) {
+ this.sessionPoolOptions.blockedStatusCodes = sessionPoolOptions.blockedStatusCodes ?? [];
+ if (this.sessionPoolOptions.blockedStatusCodes.length !== 0) {
+ log.warning(
+ `Both 'blockedStatusCodes' and 'retryOnBlocked' are set. Please note that the 'retryOnBlocked' feature might not work as expected.`,
+ );
+ }
}
- this.requestManager = requestManager;
- this.requestQueue = requestManager as RequestProvider; // TODO(v4) - the cast is not fully legitimate here, but it's fine for internal usage by the BasicCrawler
- } else {
- this.requestList = requestList;
- this.requestQueue = requestQueue;
- }
-
- this.httpClient = httpClient ?? new GotScrapingHttpClient();
- this.proxyConfiguration = proxyConfiguration;
- this.log = log;
- this.statusMessageLoggingInterval = statusMessageLoggingInterval;
- this.statusMessageCallback = statusMessageCallback as StatusMessageCallback;
- this.events = config.getEventManager();
- this.domainAccessedTime = new Map();
- this.experiments = experiments;
- this.robotsTxtFileCache = new LruCache({ maxLength: 1000 });
- this.handleSkippedRequest = this.handleSkippedRequest.bind(this);
-
- this.requestHandler = requestHandler ?? this.router;
- this.failedRequestHandler = failedRequestHandler;
- this.errorHandler = errorHandler;
-
- if (requestHandlerTimeoutSecs) {
- this.requestHandlerTimeoutMillis = requestHandlerTimeoutSecs * 1000;
- } else {
- this.requestHandlerTimeoutMillis = 60_000;
- }
-
- this.retryOnBlocked = retryOnBlocked;
- this.respectRobotsTxtFile = respectRobotsTxtFile;
- this.onSkippedRequest = onSkippedRequest;
-
- const tryEnv = (val?: string) => (val == null ? null : +val);
- // allow at least 5min for internal timeouts
- this.internalTimeoutMillis =
- tryEnv(process.env.CRAWLEE_INTERNAL_TIMEOUT) ?? Math.max(this.requestHandlerTimeoutMillis * 2, 300e3);
-
- // override the default internal timeout of request queue to respect `requestHandlerTimeoutMillis`
- if (this.requestQueue) {
- this.requestQueue.internalTimeoutMillis = this.internalTimeoutMillis;
- // for request queue v2, we want to lock requests for slightly longer than the request handler timeout so that there is some padding for locking-related overhead,
- // but never for less than a minute
- this.requestQueue.requestLockSecs = Math.max(this.requestHandlerTimeoutMillis / 1000 + 5, 60);
- }
-
- this.maxRequestRetries = maxRequestRetries;
- this.maxCrawlDepth = maxCrawlDepth;
- this.sameDomainDelayMillis = sameDomainDelaySecs * 1000;
- this.maxSessionRotations = maxSessionRotations;
- this.stats = new Statistics({
- logMessage: `${log.getOptions().prefix} request statistics:`,
- log,
- config,
- ...(this.hasExplicitId ? { id: this.crawlerId } : {}),
- ...statisticsOptions,
- });
- this.sessionPoolOptions = {
- ...sessionPoolOptions,
- log,
- };
- if (this.retryOnBlocked) {
- this.sessionPoolOptions.blockedStatusCodes = sessionPoolOptions.blockedStatusCodes ?? [];
- if (this.sessionPoolOptions.blockedStatusCodes.length !== 0) {
+ this.useSessionPool = useSessionPool;
+
+ const maxSignedInteger = 2 ** 31 - 1;
+ if (this.requestHandlerTimeoutMillis > maxSignedInteger) {
log.warning(
- `Both 'blockedStatusCodes' and 'retryOnBlocked' are set. Please note that the 'retryOnBlocked' feature might not work as expected.`,
+ `requestHandlerTimeoutMillis ${this.requestHandlerTimeoutMillis}` +
+ ` does not fit a signed 32-bit integer. Limiting the value to ${maxSignedInteger}`,
);
- }
- }
- this.useSessionPool = useSessionPool;
-
- const maxSignedInteger = 2 ** 31 - 1;
- if (this.requestHandlerTimeoutMillis > maxSignedInteger) {
- log.warning(
- `requestHandlerTimeoutMillis ${this.requestHandlerTimeoutMillis}` +
- ` does not fit a signed 32-bit integer. Limiting the value to ${maxSignedInteger}`,
- );
- this.requestHandlerTimeoutMillis = maxSignedInteger;
- }
+ this.requestHandlerTimeoutMillis = maxSignedInteger;
+ }
- this.internalTimeoutMillis = Math.min(this.internalTimeoutMillis, maxSignedInteger);
+ this.internalTimeoutMillis = Math.min(this.internalTimeoutMillis, maxSignedInteger);
- this.maxRequestsPerCrawl = maxRequestsPerCrawl;
+ this.maxRequestsPerCrawl = maxRequestsPerCrawl;
- const isMaxPagesExceeded = () =>
- this.maxRequestsPerCrawl && this.maxRequestsPerCrawl <= this.handledRequestsCount;
+ const isMaxPagesExceeded = () =>
+ this.maxRequestsPerCrawl && this.maxRequestsPerCrawl <= this.handledRequestsCount;
- // eslint-disable-next-line prefer-const
- let { isFinishedFunction, isTaskReadyFunction } = autoscaledPoolOptions;
+ // eslint-disable-next-line prefer-const
+ let { isFinishedFunction, isTaskReadyFunction } = autoscaledPoolOptions;
- // override even if `isFinishedFunction` provided by user - `keepAlive` has higher priority
- if (keepAlive) {
- isFinishedFunction = async () => false;
- }
+ // override even if `isFinishedFunction` provided by user - `keepAlive` has higher priority
+ if (keepAlive) {
+ isFinishedFunction = async () => false;
+ }
- const basicCrawlerAutoscaledPoolConfiguration: Partial<AutoscaledPoolOptions> = {
- minConcurrency: minConcurrency ?? autoscaledPoolOptions?.minConcurrency,
- maxConcurrency: maxConcurrency ?? autoscaledPoolOptions?.maxConcurrency,
- maxTasksPerMinute: maxRequestsPerMinute ?? autoscaledPoolOptions?.maxTasksPerMinute,
- runTaskFunction: this._runTaskFunction.bind(this),
- isTaskReadyFunction: async () => {
- if (isMaxPagesExceeded()) {
- this.logOncePerRun(
- 'shuttingDown',
- 'Crawler reached the maxRequestsPerCrawl limit of ' +
- `${this.maxRequestsPerCrawl} requests and will shut down soon. Requests that are in progress will be allowed to finish.`,
- );
- return false;
- }
+ const basicCrawlerAutoscaledPoolConfiguration: Partial<AutoscaledPoolOptions> = {
+ minConcurrency: minConcurrency ?? autoscaledPoolOptions?.minConcurrency,
+ maxConcurrency: maxConcurrency ?? autoscaledPoolOptions?.maxConcurrency,
+ maxTasksPerMinute: maxRequestsPerMinute ?? autoscaledPoolOptions?.maxTasksPerMinute,
+ runTaskFunction: this._runTaskFunction.bind(this),
+ isTaskReadyFunction: async () => {
+ if (isMaxPagesExceeded()) {
+ this.logOncePerRun(
+ 'shuttingDown',
+ 'Crawler reached the maxRequestsPerCrawl limit of ' +
+ `${this.maxRequestsPerCrawl} requests and will shut down soon. Requests that are in progress will be allowed to finish.`,
+ );
+ return false;
+ }
- if (this.unexpectedStop) {
- this.logOncePerRun(
- 'shuttingDown',
- 'No new requests are allowed because the `stop()` method has been called. ' +
- 'Ongoing requests will be allowed to complete.',
- );
- return false;
- }
+ if (this.unexpectedStop) {
+ this.logOncePerRun(
+ 'shuttingDown',
+ 'No new requests are allowed because the `stop()` method has been called. ' +
+ 'Ongoing requests will be allowed to complete.',
+ );
+ return false;
+ }
- return isTaskReadyFunction ? await isTaskReadyFunction() : await this._isTaskReadyFunction();
- },
- isFinishedFunction: async () => {
- if (isMaxPagesExceeded()) {
- log.info(
- `Earlier, the crawler reached the maxRequestsPerCrawl limit of ${this.maxRequestsPerCrawl} requests ` +
- 'and all requests that were in progress at that time have now finished. ' +
- `In total, the crawler processed ${this.handledRequestsCount} requests and will shut down.`,
- );
- return true;
- }
+ return isTaskReadyFunction ? await isTaskReadyFunction() : await this._isTaskReadyFunction();
+ },
+ isFinishedFunction: async () => {
+ if (isMaxPagesExceeded()) {
+ log.info(
+ `Earlier, the crawler reached the maxRequestsPerCrawl limit of ${this.maxRequestsPerCrawl} requests ` +
+ 'and all requests that were in progress at that time have now finished. ' +
+ `In total, the crawler processed ${this.handledRequestsCount} requests and will shut down.`,
+ );
+ return true;
+ }
- if (this.unexpectedStop) {
- this.log.info(
- 'The crawler has finished all the remaining ongoing requests and will shut down now.',
- );
- return true;
- }
+ if (this.unexpectedStop) {
+ this.log.info(
+ 'The crawler has finished all the remaining ongoing requests and will shut down now.',
+ );
+ return true;
+ }
- const isFinished = isFinishedFunction
- ? await isFinishedFunction()
- : await this._defaultIsFinishedFunction();
+ const isFinished = isFinishedFunction
+ ? await isFinishedFunction()
+ : await this._defaultIsFinishedFunction();
- if (isFinished) {
- const reason = isFinishedFunction
- ? "Crawler's custom isFinishedFunction() returned true, the crawler will shut down."
- : 'All requests from the queue have been processed, the crawler will shut down.';
- log.info(reason);
- }
+ if (isFinished) {
+ const reason = isFinishedFunction
+ ? "Crawler's custom isFinishedFunction() returned true, the crawler will shut down."
+ : 'All requests from the queue have been processed, the crawler will shut down.';
+ log.info(reason);
+ }
- return isFinished;
- },
- log,
- };
+ return isFinished;
+ },
+ log,
+ };
- this.autoscaledPoolOptions = { ...autoscaledPoolOptions, ...basicCrawlerAutoscaledPoolConfiguration };
+ this.autoscaledPoolOptions = { ...autoscaledPoolOptions, ...basicCrawlerAutoscaledPoolConfiguration };
+ } finally {
+ serviceLocatorScope.exitScope();
+ }
}
/**
@@ -927,7 +974,7 @@ export class BasicCrawler<
options.isStatusMessageTerminal != null ? { terminal: options.isStatusMessageTerminal } : undefined;
this.log.internal(LogLevel[(options.level as 'DEBUG') ?? 'DEBUG'], message, data);
- const client = this.config.getStorageClient();
+ const client = serviceLocator.getStorageClient();
if (!client.setStatusMessage) {
return;
@@ -1031,8 +1078,8 @@ export class BasicCrawler<
await purgeDefaultStorages({
onlyPurgeOnce: true,
- client: this.config.getStorageClient(),
- config: this.config,
+ client: serviceLocator.getStorageClient(),
+ config: serviceLocator.getConfiguration(),
});
if (requests) {
@@ -1056,8 +1103,9 @@ export class BasicCrawler<
// Attach a listener to handle migration and aborting events gracefully.
const boundPauseOnMigration = this._pauseOnMigration.bind(this);
process.once('SIGINT', sigintHandler);
- this.events.on(EventType.MIGRATING, boundPauseOnMigration);
- this.events.on(EventType.ABORTING, boundPauseOnMigration);
+ const eventManager = serviceLocator.getEventManager();
+ eventManager.on(EventType.MIGRATING, boundPauseOnMigration);
+ eventManager.on(EventType.ABORTING, boundPauseOnMigration);
let stats = {} as FinalStatistics;
@@ -1068,8 +1116,8 @@ export class BasicCrawler<
await this.teardown();
process.off('SIGINT', sigintHandler);
- this.events.off(EventType.MIGRATING, boundPauseOnMigration);
- this.events.off(EventType.ABORTING, boundPauseOnMigration);
+ eventManager.off(EventType.MIGRATING, boundPauseOnMigration);
+ eventManager.off(EventType.ABORTING, boundPauseOnMigration);
const finalStats = this.stats.calculate();
stats = {
@@ -1091,7 +1139,7 @@ export class BasicCrawler<
});
}
- const client = this.config.getStorageClient();
+ const client = serviceLocator.getStorageClient();
if (client.teardown) {
let finished = false;
@@ -1207,7 +1255,7 @@ export class BasicCrawler<
}
async useState<State extends Dictionary = Dictionary>(defaultValue = {} as State): Promise<State> {
- const kvs = await KeyValueStore.open(null, { config: this.config });
+ const kvs = await KeyValueStore.open(null, { config: serviceLocator.getConfiguration() });
if (this.hasExplicitId) {
const stateKey = `${BasicCrawler.CRAWLEE_STATE_KEY}_${this.crawlerId}`;
@@ -1373,7 +1421,7 @@ export class BasicCrawler<
* Retrieves the specified {@apilink Dataset}, or the default crawler {@apilink Dataset}.
*/
async getDataset(idOrName?: string): Promise<Dataset> {
- return Dataset.open(idOrName, { config: this.config });
+ return Dataset.open(idOrName, { config: serviceLocator.getConfiguration() });
}
/**
@@ -1443,15 +1491,20 @@ export class BasicCrawler<
* Initializes the crawler.
*/
protected async _init(): Promise<void> {
- if (!this.events.isInitialized()) {
- await this.events.init();
+ const eventManager = serviceLocator.getEventManager();
+
+ if (!eventManager.isInitialized()) {
+ await eventManager.init();
this._ownsEventManager = true;
}
- this.autoscaledPool = new AutoscaledPool(this.autoscaledPoolOptions, this.config);
+ // Initialize AutoscaledPool before awaiting _loadHandledRequestCount(),
+ // so that the caller can get a reference to it before awaiting the promise returned from run()
+ // (otherwise there would be no way)
+ this.autoscaledPool = new AutoscaledPool(this.autoscaledPoolOptions);
if (this.useSessionPool) {
- this.sessionPool = await SessionPool.open(this.sessionPoolOptions, this.config);
+ this.sessionPool = await SessionPool.open(this.sessionPoolOptions);
// Assuming there are not more than 20 browsers running at once;
this.sessionPool.setMaxListeners(20);
}
@@ -1709,7 +1762,8 @@ export class BasicCrawler<
pushData: this.pushData.bind(this),
useState: this.useState.bind(this),
sendRequest: createSendRequest(this.httpClient, request!, session),
- getKeyValueStore: async (idOrName?: string) => KeyValueStore.open(idOrName, { config: this.config }),
+ getKeyValueStore: async (idOrName?: string) =>
+ KeyValueStore.open(idOrName, { config: serviceLocator.getConfiguration() }),
registerDeferredCleanup: (cleanup) => {
deferredCleanup.push(cleanup);
},
@@ -2102,7 +2156,7 @@ export class BasicCrawler<
// the final persistence event after the crawler-specific state has been
// saved. External event managers still need an explicit event here.
if (!this._ownsEventManager) {
- this.events.emit(EventType.PERSIST_STATE, { isMigrating: false });
+ serviceLocator.getEventManager().emit(EventType.PERSIST_STATE, { isMigrating: false });
}
if (this.useSessionPool) {
@@ -2110,7 +2164,7 @@ export class BasicCrawler<
}
if (this._ownsEventManager) {
- await this.events.close();
+ await serviceLocator.getEventManager().close();
}
await this.autoscaledPool?.abort();
@@ -2135,10 +2189,10 @@ export class BasicCrawler<
this._experimentWarnings.requestLocking = true;
}
- return RequestQueueV1.open(null, { config: this.config });
+ return RequestQueueV1.open(null, { config: serviceLocator.getConfiguration() });
}
- return RequestQueue.open(null, { config: this.config });
+ return RequestQueue.open(null, { config: serviceLocator.getConfiguration() });
}
private requestMatchesEnqueueStrategy(request: Request) {
diff --git a/packages/browser-crawler/src/internals/browser-crawler.ts b/packages/browser-crawler/src/internals/browser-crawler.ts
index 98c8ef518..a8871583f 100644
--- a/packages/browser-crawler/src/internals/browser-crawler.ts
+++ b/packages/browser-crawler/src/internals/browser-crawler.ts
@@ -16,7 +16,6 @@ import type {
import {
BasicCrawler,
BLOCKED_STATUS_CODES as DEFAULT_BLOCKED_STATUS_CODES,
- Configuration,
ContextPipeline,
cookieStringToToughCookie,
enqueueLinks,
@@ -347,7 +346,6 @@ export abstract class BrowserCrawler<
> & {
contextPipelineBuilder: () => ContextPipeline<CrawlingContext, Context>;
},
- override readonly config = Configuration.getGlobalConfig(),
) {
ow(options, 'BrowserCrawlerOptions', ow.object.exactShape(BrowserCrawler.optionsShape));
const {
@@ -366,18 +364,15 @@ export abstract class BrowserCrawler<
...basicCrawlerOptions
} = options;
- super(
- {
- ...basicCrawlerOptions,
- contextPipelineBuilder: () =>
- contextPipelineBuilder()
- .compose({ action: this.performNavigation.bind(this) })
- .compose({ action: this.handleBlockedRequestByContent.bind(this) })
- .compose({ action: this.restoreRequestState.bind(this) }),
- extendContext: extendContext as (context: Context) => Awaitable<ContextExtension>,
- },
- config,
- );
+ super({
+ ...basicCrawlerOptions,
+ contextPipelineBuilder: () =>
+ contextPipelineBuilder()
+ .compose({ action: this.performNavigation.bind(this) })
+ .compose({ action: this.handleBlockedRequestByContent.bind(this) })
+ .compose({ action: this.restoreRequestState.bind(this) }),
+ extendContext: extendContext as (context: Context) => Awaitable<ContextExtension>,
+ });
// Cookies should be persisted per session only if session pool is used
if (!this.useSessionPool && persistCookiesPerSession) {
diff --git a/packages/cheerio-crawler/src/internals/cheerio-crawler.ts b/packages/cheerio-crawler/src/internals/cheerio-crawler.ts
index abe9c6725..177360f40 100644
--- a/packages/cheerio-crawler/src/internals/cheerio-crawler.ts
+++ b/packages/cheerio-crawler/src/internals/cheerio-crawler.ts
@@ -1,6 +1,5 @@
import type {
BasicCrawlingContext,
- Configuration,
EnqueueLinksOptions,
ErrorHandler,
GetUserDataFromRequest,
@@ -179,19 +178,16 @@ export class CheerioCrawler<
/**
* All `CheerioCrawler` parameters are passed via an options object.
*/
- constructor(options?: CheerioCrawlerOptions<ContextExtension, ExtendedContext>, config?: Configuration) {
- super(
- {
- ...options,
- contextPipelineBuilder: () =>
- this.buildContextPipeline()
- .compose({
- action: async (context) => await this.parseContent(context),
- })
- .compose({ action: async (context) => await this.addHelpers(context) }),
- },
- config,
- );
+ constructor(options?: CheerioCrawlerOptions<ContextExtension, ExtendedContext>) {
+ super({
+ ...options,
+ contextPipelineBuilder: () =>
+ this.buildContextPipeline()
+ .compose({
+ action: async (context) => await this.parseContent(context),
+ })
+ .compose({ action: async (context) => await this.addHelpers(context) }),
+ });
}
private async parseContent(crawlingContext: InternalHttpCrawlingContext) {
diff --git a/packages/core/src/autoscaling/autoscaled_pool.ts b/packages/core/src/autoscaling/autoscaled_pool.ts
index 926fd8604..dae34407a 100644
--- a/packages/core/src/autoscaling/autoscaled_pool.ts
+++ b/packages/core/src/autoscaling/autoscaled_pool.ts
@@ -5,7 +5,6 @@ import { addTimeoutToPromise } from '@apify/timeout';
import type { BetterIntervalID } from '@apify/utilities';
import { betterClearInterval, betterSetInterval } from '@apify/utilities';
-import { Configuration } from '../configuration.js';
import { CriticalError } from '../errors.js';
import { log as defaultLog } from '../log.js';
import type { LoadSignal } from './load_signal.js';
@@ -216,10 +215,7 @@ export class AutoscaledPool {
private tasksDonePerSecondInterval?: BetterIntervalID;
private _tasksPerMinute: number[] = Array.from({ length: 60 }, () => 0);
- constructor(
- options: AutoscaledPoolOptions,
- private readonly config = Configuration.getGlobalConfig(),
- ) {
+ constructor(options: AutoscaledPoolOptions) {
ow(
options,
ow.object.exactShape({
@@ -295,10 +291,7 @@ export class AutoscaledPool {
ssoCopy.snapshotter ??= new Snapshotter({
...snapshotterOptions,
log: this.log,
- config: this.config,
- client: this.config.getStorageClient(),
});
- ssoCopy.config ??= this.config;
this.snapshotter = ssoCopy.snapshotter;
this.loadSignals = ssoCopy.loadSignals ?? [];
this.systemStatus = new SystemStatus(ssoCopy);
diff --git a/packages/core/src/autoscaling/snapshotter.ts b/packages/core/src/autoscaling/snapshotter.ts
index 7cf3b50bc..884721600 100644
--- a/packages/core/src/autoscaling/snapshotter.ts
+++ b/packages/core/src/autoscaling/snapshotter.ts
@@ -3,9 +3,9 @@ import ow from 'ow';
import type { Log } from '@apify/log';
-import { Configuration } from '../configuration.js';
-import type { EventManager } from '../events/event_manager.js';
+import type { Configuration } from '../configuration.js';
import { log as defaultLog } from '../log.js';
+import { serviceLocator } from '../service_locator.js';
import type { ClientLoadSignal, ClientSnapshot } from './client_load_signal.js';
import { createClientLoadSignal } from './client_load_signal.js';
import type { CpuLoadSignal, CpuSnapshot } from './cpu_load_signal.js';
@@ -99,7 +99,6 @@ export class Snapshotter {
log: Log;
client: StorageClient;
config: Configuration;
- events: EventManager;
private readonly memorySignal: MemoryLoadSignal;
private readonly eventLoopSignal: EventLoopLoadSignal;
@@ -158,14 +157,13 @@ export class Snapshotter {
maxUsedMemoryRatio = 0.9,
maxClientErrors = 3,
log = defaultLog,
- config = Configuration.getGlobalConfig(),
- client = config.getStorageClient(),
+ config = serviceLocator.getConfiguration(),
+ client = serviceLocator.getStorageClient(),
} = options;
this.log = log.child({ prefix: 'Snapshotter' });
this.client = client;
this.config = config;
- this.events = this.config.getEventManager();
const snapshotHistoryMillis = snapshotHistorySecs * 1000;
diff --git a/packages/core/src/autoscaling/system_status.ts b/packages/core/src/autoscaling/system_status.ts
index 5caf69272..cfbd73259 100644
--- a/packages/core/src/autoscaling/system_status.ts
+++ b/packages/core/src/autoscaling/system_status.ts
@@ -1,6 +1,5 @@
import ow from 'ow';
-import type { Configuration } from '../configuration.js';
import type { LoadSignal } from './load_signal.js';
import { evaluateLoadSignalSample } from './load_signal.js';
import { Snapshotter } from './snapshotter.js';
@@ -87,9 +86,6 @@ export interface SystemStatusOptions {
* considered overloaded.
*/
loadSignals?: LoadSignal[];
-
- /** @internal */
- config?: Configuration;
}
export interface ClientInfo {
@@ -159,7 +155,6 @@ export class SystemStatus {
maxClientOverloadedRatio: ow.optional.number,
snapshotter: ow.optional.object,
loadSignals: ow.optional.array,
- config: ow.optional.object,
}),
);
@@ -171,11 +166,10 @@ export class SystemStatus {
maxClientOverloadedRatio = 0.3,
snapshotter,
loadSignals = [],
- config,
} = options;
this.currentHistoryMillis = currentHistorySecs * 1000;
- this.snapshotter = snapshotter || new Snapshotter({ config });
+ this.snapshotter = snapshotter || new Snapshotter();
// Built-in signals from the snapshotter + any custom signals
this.signals = [...this.snapshotter.getLoadSignals(), ...loadSignals];
diff --git a/packages/core/src/configuration.ts b/packages/core/src/configuration.ts
index 7155efe3a..c68ae1765 100644
--- a/packages/core/src/configuration.ts
+++ b/packages/core/src/configuration.ts
@@ -1,41 +1,16 @@
-import { AsyncLocalStorage } from 'node:async_hooks';
import { EventEmitter } from 'node:events';
import { readFileSync } from 'node:fs';
import { join } from 'node:path';
-import type { MemoryStorageOptions } from '@crawlee/memory-storage';
-import { MemoryStorage } from '@crawlee/memory-storage';
-import type { Dictionary, StorageClient } from '@crawlee/types';
+import type { Dictionary } from '@crawlee/types';
import { pathExistsSync } from 'fs-extra/esm';
import log, { LogLevel } from '@apify/log';
-import { type EventManager } from './events/event_manager.js';
-import { LocalEventManager } from './events/local_event_manager.js';
-import type { StorageManager } from './storages/storage_manager.js';
-import { type Constructor, entries } from './typedefs.js';
-
-const memoryStorages = new Map<string, MemoryStorage>();
+import { serviceLocator } from './service_locator.js';
+import { entries } from './typedefs.js';
export interface ConfigurationOptions {
- /**
- * Defines storage client to be used.
- * @default {@apilink MemoryStorage}
- */
- storageClient?: StorageClient;
-
- /**
- * Defines the Event Manager to be used.
- * @default {@apilink EventManager}
- */
- eventManager?: EventManager;
-
- /**
- * Could be used to adjust the storage client behavior
- * e.g. {@apilink MemoryStorageOptions} could be used to adjust the {@apilink MemoryStorage} behavior.
- */
- storageClientOptions?: Dictionary;
-
/**
* Default dataset id.
*
@@ -203,7 +178,7 @@ export interface ConfigurationOptions {
* // Create a new configuration
* const config = new Configuration({ persistStateIntervalMillis: 30_000 });
* // Pass the configuration to the crawler
- * const crawler = new BasicCrawler({ ... }, config);
+ * const crawler = new BasicCrawler({ configuration: config, ... });
* ```
*
* The configuration provided via environment variables always takes precedence. We can also
@@ -291,19 +266,7 @@ export class Configuration {
persistStorage: true,
};
- /**
- * Provides access to the current-instance-scoped Configuration without passing it around in parameters.
- * @internal
- */
- static storage = new AsyncLocalStorage<Configuration>();
-
protected options!: Map<keyof ConfigurationOptions, ConfigurationOptions[keyof ConfigurationOptions]>;
- protected services = new Map<string, unknown>();
-
- /** @internal */
- static globalConfig?: Configuration;
-
- public readonly storageManagers = new Map<Constructor, StorageManager>();
/**
* Creates new `Configuration` instance with provided options. Env vars will have precedence over those.
@@ -385,117 +348,13 @@ export class Configuration {
this.options.set(key, value);
}
- /**
- * Sets value for given option. Only affects the global `Configuration` instance, the value will not be propagated down to the env var.
- * To reset a value, we can omit the `value` argument or pass `undefined` there.
- */
- static set(key: keyof ConfigurationOptions, value?: any): void {
- this.getGlobalConfig().set(key, value);
- }
-
- /**
- * Returns cached instance of {@apilink StorageClient} using options as defined in the environment variables or in
- * this {@apilink Configuration} instance. Only first call of this method will create the client, following calls will
- * return the same client instance.
- *
- * Caching works based on the `storageClientOptions`, so calling this method with different options will return
- * multiple instances, one for each variant of the options.
- * @internal
- */
- getStorageClient(): StorageClient {
- if (this.options.has('storageClient')) {
- return this.options.get('storageClient') as StorageClient;
- }
-
- const options = this.options.get('storageClientOptions') as Dictionary;
- return this.createMemoryStorage(options);
- }
-
- getEventManager(): EventManager {
- if (this.options.has('eventManager')) {
- return this.options.get('eventManager') as EventManager;
- }
-
- if (this.services.has('eventManager')) {
- return this.services.get('eventManager') as EventManager;
- }
-
- const eventManager = new LocalEventManager(this);
- this.services.set('eventManager', eventManager);
-
- return eventManager;
- }
-
- /**
- * Creates an instance of MemoryStorage using options as defined in the environment variables or in this `Configuration` instance.
- * @internal
- */
- createMemoryStorage(options: MemoryStorageOptions = {}): MemoryStorage {
- const storageOptions = {
- persistStorage: this.get('persistStorage') as boolean,
- // Override persistStorage if user provides it via storageClientOptions
- ...options,
- };
-
- const cacheKey = `MemoryStorage-${JSON.stringify(storageOptions)}-${process.env.CRAWLEE_STORAGE_DIR ?? ''}`;
-
- // Clients writing to the same directory are not independent, so they are shared between
- // `Configuration` instances - otherwise they would purge each other's data mid-run.
- // Purely in-memory clients have no such conflict and stay isolated per configuration.
- const cache = storageOptions.persistStorage ? memoryStorages : this.services;
-
- if (!cache.has(cacheKey)) {
- cache.set(cacheKey, new MemoryStorage(storageOptions));
- }
-
- return cache.get(cacheKey) as MemoryStorage;
- }
-
- useStorageClient(client: StorageClient): void {
- this.options.set('storageClient', client);
- }
-
- static useStorageClient(client: StorageClient): void {
- this.getGlobalConfig().useStorageClient(client);
- }
-
- useEventManager(events: EventManager): void {
- this.options.set('eventManager', events);
- }
-
/**
* Returns the global configuration instance. It will respect the environment variables.
+ *
+ * Delegates to the global ServiceLocator, making it the single source of truth for service management.
*/
static getGlobalConfig(): Configuration {
- if (Configuration.storage.getStore()) {
- return Configuration.storage.getStore()!;
- }
-
- Configuration.globalConfig ??= new Configuration();
- return Configuration.globalConfig;
- }
-
- /**
- * Gets default {@apilink StorageClient} instance.
- */
- static getStorageClient(): StorageClient {
- return this.getGlobalConfig().getStorageClient();
- }
-
- /**
- * Gets default {@apilink EventManager} instance.
- */
- static getEventManager(): EventManager {
- return this.getGlobalConfig().getEventManager();
- }
-
- /**
- * Resets global configuration instance. The default instance holds configuration based on env vars,
- * if we want to change them, we need to first reset the global state. Used mainly for testing purposes.
- */
- static resetGlobalState(): void {
- delete this.globalConfig;
- memoryStorages.clear();
+ return serviceLocator.getConfiguration();
}
protected buildOptions(options: ConfigurationOptions) {
diff --git a/packages/core/src/crawlers/statistics.ts b/packages/core/src/crawlers/statistics.ts
index 485d6738a..dfbfb77a9 100644
--- a/packages/core/src/crawlers/statistics.ts
+++ b/packages/core/src/crawlers/statistics.ts
@@ -2,10 +2,10 @@ import ow from 'ow';
import type { Log } from '@apify/log';
-import { Configuration } from '../configuration.js';
import type { EventManager } from '../events/event_manager.js';
import { EventType } from '../events/event_manager.js';
import { log as defaultLog } from '../log.js';
+import { serviceLocator } from '../service_locator.js';
import { KeyValueStore } from '../storages/key_value_store.js';
import { ErrorTracker } from './error_tracker.js';
@@ -84,11 +84,6 @@ export class Statistics {
*/
readonly requestRetryHistogram: number[] = [];
- /**
- * Contains the associated Configuration instance
- */
- private readonly config: Configuration;
-
protected keyValueStore?: KeyValueStore = undefined;
protected persistStateKey: string;
private logIntervalMillis: number;
@@ -98,9 +93,16 @@ export class Statistics {
private readonly log: Log;
private instanceStart!: number;
private logInterval: unknown;
- private events: EventManager;
+ private _events?: EventManager;
private persistenceOptions: PersistenceOptions;
+ private get events(): EventManager {
+ if (!this._events) {
+ this._events = serviceLocator.getEventManager();
+ }
+ return this._events;
+ }
+
/**
* @internal
*/
@@ -112,7 +114,6 @@ export class Statistics {
logMessage: ow.optional.string,
log: ow.optional.object,
keyValueStore: ow.optional.object,
- config: ow.optional.object,
persistenceOptions: ow.optional.object,
saveErrorSnapshots: ow.optional.boolean,
id: ow.optional.any(ow.number, ow.string),
@@ -123,7 +124,6 @@ export class Statistics {
logIntervalSecs = 60,
logMessage = 'Statistics',
keyValueStore,
- config = Configuration.getGlobalConfig(),
persistenceOptions = {
enable: true,
},
@@ -141,8 +141,6 @@ export class Statistics {
this.logMessage = logMessage;
this.keyValueStore = keyValueStore;
this.listener = this.persistState.bind(this);
- this.events = config.getEventManager();
- this.config = config;
this.persistenceOptions = persistenceOptions;
// initialize by "resetting"
@@ -282,7 +280,7 @@ export class Statistics {
* displaying the current state in predefined intervals
*/
async startCapturing() {
- this.keyValueStore ??= await KeyValueStore.open(null, { config: this.config });
+ this.keyValueStore ??= await KeyValueStore.open(null, { config: serviceLocator.getConfiguration() });
if (this.state.crawlerStartedAt === null) {
this.state.crawlerStartedAt = new Date();
@@ -335,7 +333,7 @@ export class Statistics {
this.log.debug('Persisting state', { persistStateKey: this.persistStateKey });
// use half the interval of `persistState` to avoid race conditions
- const persistStateIntervalMillis = this.config.get('persistStateIntervalMillis')!;
+ const persistStateIntervalMillis = serviceLocator.getConfiguration().get('persistStateIntervalMillis')!;
const timeoutSecs = persistStateIntervalMillis / 2_000;
await this.keyValueStore
.setValue(this.persistStateKey, this.toJSON(), {
@@ -394,7 +392,9 @@ export class Statistics {
protected _teardown(): void {
// this can be called before a call to startCapturing happens (or in a 'finally' block)
- this.events.off(EventType.PERSIST_STATE, this.listener);
+ // Only unsubscribe if event manager was already resolved — avoid eagerly resolving it
+ // (e.g. during the constructor's reset() call, which would capture the wrong context)
+ this._events?.off(EventType.PERSIST_STATE, this.listener);
if (this.logInterval) {
clearInterval(this.logInterval as number);
@@ -463,12 +463,6 @@ export interface StatisticsOptions {
*/
keyValueStore?: KeyValueStore;
- /**
- * Configuration instance to use
- * @default Configuration.getGlobalConfig()
- */
- config?: Configuration;
-
/**
* Control how and when to persist the statistics.
*/
diff --git a/packages/core/src/errors.ts b/packages/core/src/errors.ts
index 0dd551944..1acee0887 100644
--- a/packages/core/src/errors.ts
+++ b/packages/core/src/errors.ts
@@ -85,3 +85,15 @@ export class RequestHandlerError extends Error {
super(undefined, { cause: error, ...options });
}
}
+
+/**
+ * Thrown when attempting to set a different service instance after one has already been retrieved.
+ */
+export class ServiceConflictError extends Error {
+ constructor(serviceName: string, newValue: unknown, existingValue: unknown) {
+ super(
+ `Service ${serviceName} is already in use. ` +
+ `Existing value: ${existingValue}, attempted new value: ${newValue}.`,
+ );
+ }
+}
diff --git a/packages/core/src/events/event_manager.ts b/packages/core/src/events/event_manager.ts
index e134974ec..399bd519b 100644
--- a/packages/core/src/events/event_manager.ts
+++ b/packages/core/src/events/event_manager.ts
@@ -4,7 +4,10 @@ import log from '@apify/log';
import type { BetterIntervalID } from '@apify/utilities';
import { betterClearInterval, betterSetInterval } from '@apify/utilities';
-import { Configuration } from '../configuration.js';
+export interface EventManagerOptions {
+ /** Interval between emitted `persistState` events in milliseconds. */
+ persistStateIntervalMillis: number;
+}
export const enum EventType {
PERSIST_STATE = 'persistState',
@@ -26,13 +29,15 @@ export abstract class EventManager {
protected initialized = false;
protected intervals: Intervals = {};
protected log = log.child({ prefix: 'Events' });
+ private persistStateIntervalMillis: number;
- constructor(readonly config = Configuration.getGlobalConfig()) {
+ constructor(options: EventManagerOptions) {
+ this.persistStateIntervalMillis = options.persistStateIntervalMillis;
this.events.setMaxListeners(50);
}
/**
- * Initializes the event manager by creating the `persistState` event interval.
+ * Initializes the event manager by starting the `persistState` event interval.
* This is automatically called at the beginning of `crawler.run()`.
*/
async init() {
@@ -40,11 +45,11 @@ export abstract class EventManager {
return;
}
- const persistStateIntervalMillis = this.config.get('persistStateIntervalMillis')!;
this.intervals.persistState = betterSetInterval((intervalCallback: () => unknown) => {
this.emit(EventType.PERSIST_STATE, { isMigrating: false });
intervalCallback();
- }, persistStateIntervalMillis);
+ }, this.persistStateIntervalMillis);
+
this.initialized = true;
}
diff --git a/packages/core/src/events/local_event_manager.ts b/packages/core/src/events/local_event_manager.ts
index 88baa0b99..3063d4e21 100644
--- a/packages/core/src/events/local_event_manager.ts
+++ b/packages/core/src/events/local_event_manager.ts
@@ -1,14 +1,39 @@
-import { getCurrentCpuTicksV2, getMemoryInfo, isContainerized } from '@crawlee/utils';
-
import log from '@apify/log';
import { betterClearInterval, betterSetInterval } from '@apify/utilities';
import type { SystemInfo } from '../autoscaling/system_status.js';
-import { EventManager, EventType } from './event_manager.js';
+import type { Configuration } from '../configuration.js';
+import { serviceLocator } from '../service_locator.js';
+import { EventManager, type EventManagerOptions, EventType } from './event_manager.js';
+
+export interface LocalEventManagerOptions extends EventManagerOptions {
+ /** Interval between emitted `systemInfo` events in milliseconds. */
+ systemInfoIntervalMillis: number;
+}
export class LocalEventManager extends EventManager {
+ private systemInfoIntervalMillis: number;
+
+ constructor(options: LocalEventManagerOptions) {
+ super(options);
+ this.systemInfoIntervalMillis = options.systemInfoIntervalMillis;
+ }
+
+ /**
+ * Creates a new `LocalEventManager` based on the provided `Configuration`.
+ * Uses the global configuration from the service locator if none is provided.
+ */
+ static fromConfig(config?: Configuration): LocalEventManager {
+ const resolvedConfig = config ?? serviceLocator.getConfiguration();
+
+ return new LocalEventManager({
+ persistStateIntervalMillis: resolvedConfig.get('persistStateIntervalMillis'),
+ systemInfoIntervalMillis: resolvedConfig.get('systemInfoIntervalMillis'),
+ });
+ }
+
/**
- * Initializes the EventManager and sets up periodic `systemInfo` and `persistState` events.
+ * Initializes the EventManager and sets up periodic `systemInfo` events.
* This is automatically called at the beginning of `crawler.run()`.
*/
override async init() {
@@ -18,9 +43,11 @@ export class LocalEventManager extends EventManager {
await super.init();
- const systemInfoIntervalMillis = this.config.get('systemInfoIntervalMillis')!;
this.emitSystemInfoEvent = this.emitSystemInfoEvent.bind(this);
- this.intervals.systemInfo = betterSetInterval(this.emitSystemInfoEvent.bind(this), systemInfoIntervalMillis);
+ this.intervals.systemInfo = betterSetInterval(
+ this.emitSystemInfoEvent.bind(this),
+ this.systemInfoIntervalMillis,
+ );
}
/**
@@ -40,7 +67,7 @@ export class LocalEventManager extends EventManager {
*/
async emitSystemInfoEvent(intervalCallback: () => unknown) {
const info = await this.createSystemInfo({
- maxUsedCpuRatio: this.config.get('maxUsedCpuRatio'),
+ maxUsedCpuRatio: serviceLocator.getConfiguration().get('maxUsedCpuRatio'),
});
this.events.emit(EventType.SYSTEM_INFO, info);
intervalCallback();
@@ -50,7 +77,8 @@ export class LocalEventManager extends EventManager {
* @internal
*/
async isContainerizedWrapper() {
- return this.config.get('containerized', await isContainerized());
+ const { isContainerized } = await import('@crawlee/utils');
+ return serviceLocator.getConfiguration().get('containerized', await isContainerized());
}
/**
@@ -65,6 +93,7 @@ export class LocalEventManager extends EventManager {
}
private async createCpuInfo(options: { maxUsedCpuRatio: number }) {
+ const { getCurrentCpuTicksV2 } = await import('@crawlee/utils');
const usedCpuRatio = await getCurrentCpuTicksV2(await this.isContainerizedWrapper());
return {
cpuCurrentUsage: usedCpuRatio * 100,
@@ -74,7 +103,7 @@ export class LocalEventManager extends EventManager {
private async createMemoryInfo() {
try {
- const memInfo = await this.getMemoryInfo(this.isContainerizedWrapper());
+ const memInfo = await this.getMemoryInfo();
return {
memTotalBytes: memInfo.totalBytes,
memCurrentBytes: memInfo.mainProcessBytes + memInfo.childProcessesBytes,
diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts
index 1f327e0e6..2661b8be6 100644
--- a/packages/core/src/index.ts
+++ b/packages/core/src/index.ts
@@ -1,6 +1,7 @@
export * from './errors.js';
export * from './autoscaling/index.js';
export * from './configuration.js';
+export * from './service_locator.js';
export * from './crawlers/index.js';
export * from './enqueue_links/index.js';
export * from './events/index.js';
diff --git a/packages/core/src/recoverable_state.ts b/packages/core/src/recoverable_state.ts
index d19d36ba8..aa6707971 100644
--- a/packages/core/src/recoverable_state.ts
+++ b/packages/core/src/recoverable_state.ts
@@ -1,4 +1,5 @@
-import { Configuration, EventType, KeyValueStore } from '@crawlee/core';
+import type { Configuration } from '@crawlee/core';
+import { EventType, KeyValueStore, serviceLocator } from '@crawlee/core';
import type { Log } from '@apify/log';
import log from '@apify/log';
@@ -81,7 +82,6 @@ export class RecoverableState<TStateModel = Record<string, unknown>> {
private readonly persistStateKvsId?: string;
private keyValueStore: KeyValueStore | null = null;
private readonly log: Log;
- private readonly config: Configuration;
private readonly serialize: (state: TStateModel) => string;
private readonly deserialize: (serializedState: string) => TStateModel;
@@ -97,7 +97,6 @@ export class RecoverableState<TStateModel = Record<string, unknown>> {
this.persistStateKvsName = options.persistStateKvsName;
this.persistStateKvsId = options.persistStateKvsId;
this.log = options.logger ?? log.child({ prefix: 'RecoverableState' });
- this.config = options.config ?? Configuration.getGlobalConfig();
this.serialize = options.serialize ?? JSON.stringify;
this.deserialize = options.deserialize ?? JSON.parse;
@@ -123,13 +122,13 @@ export class RecoverableState<TStateModel = Record<string, unknown>> {
}
this.keyValueStore = await KeyValueStore.open(this.persistStateKvsName ?? this.persistStateKvsId, {
- config: this.config,
+ config: serviceLocator.getConfiguration(),
});
await this.loadSavedState();
// Register for persist state events
- const eventManager = this.config.getEventManager();
+ const eventManager = serviceLocator.getEventManager();
eventManager.on(EventType.PERSIST_STATE, this.persistState);
return this.currentValue;
@@ -146,7 +145,7 @@ export class RecoverableState<TStateModel = Record<string, unknown>> {
return;
}
- const eventManager = this.config.getEventManager();
+ const eventManager = serviceLocator.getEventManager();
eventManager.off(EventType.PERSIST_STATE, this.persistState);
await this.persistState();
}
diff --git a/packages/core/src/service_locator.ts b/packages/core/src/service_locator.ts
new file mode 100644
index 000000000..d22b1c84d
--- /dev/null
+++ b/packages/core/src/service_locator.ts
@@ -0,0 +1,363 @@
+import { AsyncLocalStorage } from 'node:async_hooks';
+
+import { MemoryStorage } from '@crawlee/memory-storage';
+import type { StorageClient } from '@crawlee/types';
+
+import log from '@apify/log';
+
+import { Configuration } from './configuration.js';
+import { ServiceConflictError } from './errors.js';
+import type { EventManager } from './events/event_manager.js';
+import { LocalEventManager } from './events/local_event_manager.js';
+import type { IStorage, StorageManager } from './storages/storage_manager.js';
+import type { Constructor } from './typedefs.js';
+
+interface ServiceLocatorInterface {
+ /**
+ * Get the configuration.
+ * Creates a default Configuration instance if none has been set.
+ */
+ getConfiguration(): Configuration;
+
+ /**
+ * Set the configuration.
+ *
+ * @param configuration The configuration to set
+ * @throws {ServiceConflictError} If a different configuration has already been retrieved
+ */
+ setConfiguration(configuration: Configuration): void;
+
+ /**
+ * Get the event manager.
+ * Creates a default LocalEventManager instance if none has been set.
+ */
+ getEventManager(): EventManager;
+
+ /**
+ * Set the event manager.
+ *
+ * @param eventManager The event manager to set
+ * @throws {ServiceConflictError} If a different event manager has already been retrieved
+ */
+ setEventManager(eventManager: EventManager): void;
+
+ /**
+ * Get the storage client.
+ * Creates a default MemoryStorage instance if none has been set.
+ */
+ getStorageClient(): StorageClient;
+
+ /**
+ * Set the storage client.
+ *
+ * @param storageClient The storage client to set
+ * @throws {ServiceConflictError} If a different storage client has already been retrieved
+ */
+ setStorageClient(storageClient: StorageClient): void;
+
+ getStorageManager(constructor: Constructor<IStorage>): StorageManager | undefined;
+
+ setStorageManager(constructor: Constructor<IStorage>, storageManager: StorageManager): void;
+
+ /**
+ * Clears all storage manager caches.
+ * @internal
+ */
+ clearStorageManagerCache(): void;
+
+ /**
+ * Resets the service locator to its initial state.
+ * Used mainly for testing purposes.
+ * @internal
+ */
+ reset(): void;
+}
+
+/**
+ * Service locator for managing the services used by Crawlee.
+ *
+ * All services are initialized to their default value lazily.
+ *
+ * There are two primary usage patterns:
+ *
+ * **1. Global service locator (for default services):**
+ * ```typescript
+ * import { serviceLocator, BasicCrawler } from 'crawlee';
+ *
+ * // Optionally configure global services before creating crawlers
+ * serviceLocator.setStorageClient(myCustomClient);
+ *
+ * // Crawler uses global services
+ * const crawler = new BasicCrawler({ ... });
+ * ```
+ *
+ * **2. Per-crawler services (recommended for isolation):**
+ * ```typescript
+ * import { BasicCrawler, Configuration, LocalEventManager } from 'crawlee';
+ * import { MemoryStorage } from '@crawlee/memory-storage';
+ *
+ * const crawler = new BasicCrawler({
+ * requestHandler: async ({ request }) => { ... },
+ * configuration: new Configuration({ ... }), // custom config
+ * storageClient: new MemoryStorage(), // custom storage
+ * eventManager: LocalEventManager.fromConfig(), // custom events
+ * });
+ * // Crawler has its own isolated ServiceLocator instance
+ * ```
+ */
+export class ServiceLocator implements ServiceLocatorInterface {
+ private configuration?: Configuration;
+ private eventManager?: EventManager;
+ private storageClient?: StorageClient;
+
+ /**
+ * Storage managers for Dataset, KeyValueStore, and RequestQueue.
+ * Manages caching and lifecycle of storage instances.
+ */
+ private storageManagers = new Map<Constructor<IStorage>, StorageManager>();
+
+ /**
+ * Creates a new ServiceLocator instance.
+ *
+ * @param configuration Optional configuration instance to use
+ * @param eventManager Optional event manager instance to use
+ * @param storageClient Optional storage client instance to use
+ */
+ constructor(configuration?: Configuration, eventManager?: EventManager, storageClient?: StorageClient) {
+ this.configuration = configuration;
+ this.eventManager = eventManager;
+ this.storageClient = storageClient;
+ }
+
+ getConfiguration(): Configuration {
+ if (!this.configuration) {
+ log.debug('No configuration set, implicitly creating and using default Configuration.');
+ this.configuration = new Configuration();
+ }
+ return this.configuration;
+ }
+
+ setConfiguration(configuration: Configuration): void {
+ // Same instance, no need to do anything
+ if (this.configuration === configuration) {
+ return;
+ }
+
+ // Already have a different configuration that was retrieved
+ if (this.configuration) {
+ throw new ServiceConflictError('Configuration', configuration, this.configuration);
+ }
+
+ this.configuration = configuration;
+ }
+
+ getEventManager(): EventManager {
+ if (!this.eventManager) {
+ log.debug('No event manager set, implicitly creating and using default LocalEventManager.');
+ if (!this.configuration) {
+ log.warning(
+ 'Implicit creation of event manager will implicitly set configuration as side effect. ' +
+ 'It is advised to explicitly first set the configuration instead.',
+ );
+ }
+ this.eventManager = LocalEventManager.fromConfig(this.getConfiguration());
+ }
+ return this.eventManager;
+ }
+
+ setEventManager(eventManager: EventManager): void {
+ // Same instance, no need to do anything
+ if (this.eventManager === eventManager) {
+ return;
+ }
+
+ // Already have a different event manager that was retrieved
+ if (this.eventManager) {
+ throw new ServiceConflictError('EventManager', eventManager, this.eventManager);
+ }
+
+ this.eventManager = eventManager;
+ }
+
+ getStorageClient(): StorageClient {
+ if (!this.storageClient) {
+ log.debug('No storage client set, implicitly creating and using default MemoryStorage.');
+ if (!this.configuration) {
+ log.warning(
+ 'Implicit creation of storage client will implicitly set configuration as side effect. ' +
+ 'It is advised to explicitly first set the configuration instead.',
+ );
+ }
+ const config = this.getConfiguration();
+ this.storageClient = new MemoryStorage({
+ persistStorage: config.get('persistStorage'),
+ });
+ }
+ return this.storageClient;
+ }
+
+ setStorageClient(storageClient: StorageClient): void {
+ // Same instance, no need to do anything
+ if (this.storageClient === storageClient) {
+ return;
+ }
+
+ // Already have a different storage client that was retrieved
+ if (this.storageClient) {
+ throw new ServiceConflictError('StorageClient', storageClient, this.storageClient);
+ }
+
+ this.storageClient = storageClient;
+ }
+
+ getStorageManager(constructor: Constructor<IStorage>): StorageManager | undefined {
+ return this.storageManagers.get(constructor);
+ }
+
+ setStorageManager(constructor: Constructor<IStorage>, storageManager: StorageManager): void {
+ if (this.storageManagers.has(constructor)) {
+ throw new ServiceConflictError(
+ `StorageManager(${constructor.name})`,
+ storageManager,
+ this.storageManagers.get(constructor),
+ );
+ }
+
+ this.storageManagers.set(constructor, storageManager);
+ }
+
+ clearStorageManagerCache(): void {
+ this.storageManagers.forEach((manager) => {
+ // KeyValueStore has a clearCache method on its instances
+ // TODO this uses fragile string matching and `any` casts into private fields - remove as part of
+ // https://github.com/apify/crawlee/issues/3075 (Storage instance management will be reworked significantly)
+ if ((manager as any).name === 'KeyValueStore') {
+ (manager as any).cache?.forEach((item: any) => {
+ item.clearCache?.();
+ });
+ }
+ });
+ this.storageManagers.clear();
+ }
+
+ reset(): void {
+ this.configuration = undefined;
+ this.eventManager = undefined;
+ this.storageClient = undefined;
+ this.clearStorageManagerCache();
+ }
+}
+
+/**
+ * Used as the default service provider when crawlers don't specify custom services.
+ */
+const globalServiceLocator = new ServiceLocator();
+
+const serviceLocatorStorage = new AsyncLocalStorage<ServiceLocatorInterface>();
+
+/**
+ * Wraps all methods on `target` so that any code they invoke will see the given
+ * `serviceLocator` via `AsyncLocalStorage`, rather than the global one.
+ *
+ * Walks the prototype chain and replaces each method on the *instance* (not the prototype)
+ * with a wrapper that calls `serviceLocatorStorage.run(serviceLocator, originalMethod)`.
+ *
+ * The `AsyncLocalStorage` context propagates through the entire sync/async call tree of each
+ * wrapped method — including `super` calls, since the prototype methods execute within the
+ * context established by the instance-level wrapper.
+ *
+ * @internal
+ * @returns Scope control functions: `run` executes a callback within the scoped context,
+ * `enterScope`/`exitScope` allow entering/leaving the scope imperatively (e.g., for constructor bodies).
+ */
+export function bindMethodsToServiceLocator(
+ serviceLocator: ServiceLocator,
+ target: {},
+): { run: <T>(fn: () => T) => T; enterScope: () => void; exitScope: () => void } {
+ let proto = Object.getPrototypeOf(target);
+
+ while (proto !== null && proto !== Object.prototype) {
+ const propertyKeys = [...Object.getOwnPropertyNames(proto), ...Object.getOwnPropertySymbols(proto)];
+
+ for (const propertyKey of propertyKeys) {
+ const descriptor = Object.getOwnPropertyDescriptor(proto, propertyKey);
+
+ // We use property descriptors rather than accessing target[propertyKey] directly,
+ // because that would trigger getters and cause unwanted side effects.
+ // Skip getters, setters, and constructors — only wrap regular methods.
+ if (
+ propertyKey === 'constructor' ||
+ !descriptor ||
+ descriptor.get ||
+ descriptor.set ||
+ typeof descriptor.value !== 'function'
+ )
+ continue;
+
+ const original = descriptor.value;
+ (target as Record<string | symbol, unknown>)[propertyKey] = (...args: any[]) => {
+ return serviceLocatorStorage.run(serviceLocator, () => {
+ return original.apply(target, args);
+ });
+ };
+ }
+
+ proto = Object.getPrototypeOf(proto);
+ }
+
+ let previousStore: ServiceLocatorInterface | undefined;
+
+ return {
+ run: <T>(fn: () => T): T => serviceLocatorStorage.run(serviceLocator, fn),
+ enterScope: () => {
+ previousStore = serviceLocatorStorage.getStore();
+ serviceLocatorStorage.enterWith(serviceLocator);
+ },
+ exitScope: () => {
+ serviceLocatorStorage.enterWith(previousStore as any); // casting to any so that `undefined` is accepted - this "unsets" the AsyncLocalStorage
+ },
+ };
+}
+
+export const serviceLocator: ServiceLocatorInterface = {
+ getConfiguration(): Configuration {
+ const currentServiceLocator = serviceLocatorStorage.getStore() ?? globalServiceLocator;
+ return currentServiceLocator.getConfiguration();
+ },
+ setConfiguration(configuration: Configuration): void {
+ const currentServiceLocator = serviceLocatorStorage.getStore() ?? globalServiceLocator;
+ currentServiceLocator.setConfiguration(configuration);
+ },
+ getEventManager(): EventManager {
+ const currentServiceLocator = serviceLocatorStorage.getStore() ?? globalServiceLocator;
+ return currentServiceLocator.getEventManager();
+ },
+ setEventManager(eventManager: EventManager): void {
+ const currentServiceLocator = serviceLocatorStorage.getStore() ?? globalServiceLocator;
+ currentServiceLocator.setEventManager(eventManager);
+ },
+ getStorageClient(): StorageClient {
+ const currentServiceLocator = serviceLocatorStorage.getStore() ?? globalServiceLocator;
+ return currentServiceLocator.getStorageClient();
+ },
+ setStorageClient(storageClient: StorageClient): void {
+ const currentServiceLocator = serviceLocatorStorage.getStore() ?? globalServiceLocator;
+ currentServiceLocator.setStorageClient(storageClient);
+ },
+ getStorageManager(constructor: Constructor<IStorage>): StorageManager | undefined {
+ const currentServiceLocator = serviceLocatorStorage.getStore() ?? globalServiceLocator;
+ return currentServiceLocator.getStorageManager(constructor);
+ },
+ setStorageManager(constructor: Constructor<IStorage>, storageManager: StorageManager): void {
+ const currentServiceLocator = serviceLocatorStorage.getStore() ?? globalServiceLocator;
+ currentServiceLocator.setStorageManager(constructor, storageManager);
+ },
+ clearStorageManagerCache(): void {
+ const currentServiceLocator = serviceLocatorStorage.getStore() ?? globalServiceLocator;
+ currentServiceLocator.clearStorageManagerCache();
+ },
+ reset(): void {
+ const currentServiceLocator = serviceLocatorStorage.getStore() ?? globalServiceLocator;
+ currentServiceLocator.reset();
+ },
+};
diff --git a/packages/core/src/session_pool/session_pool.ts b/packages/core/src/session_pool/session_pool.ts
index 0abc5aed7..16fd728d7 100644
--- a/packages/core/src/session_pool/session_pool.ts
+++ b/packages/core/src/session_pool/session_pool.ts
@@ -6,11 +6,11 @@ import ow from 'ow';
import type { Log } from '@apify/log';
-import { Configuration } from '../configuration.js';
import type { PersistenceOptions } from '../crawlers/statistics.js';
import type { EventManager } from '../events/event_manager.js';
import { EventType } from '../events/event_manager.js';
import { log as defaultLog } from '../log.js';
+import { serviceLocator } from '../service_locator.js';
import { KeyValueStore } from '../storages/key_value_store.js';
import { BLOCKED_STATUS_CODES, MAX_POOL_SIZE, PERSIST_STATE_KEY } from './consts.js';
import type { SessionOptions } from './session.js';
@@ -155,10 +155,7 @@ export class SessionPool extends EventEmitter {
/**
* @internal
*/
- constructor(
- options: SessionPoolOptions = {},
- readonly config = Configuration.getGlobalConfig(),
- ) {
+ constructor(options: SessionPoolOptions = {}) {
super();
ow(
@@ -188,9 +185,8 @@ export class SessionPool extends EventEmitter {
},
} = options;
- this.config = config;
this.blockedStatusCodes = blockedStatusCodes;
- this.events = config.getEventManager();
+ this.events = serviceLocator.getEventManager();
this.log = log.child({ prefix: 'SessionPool' });
this.persistenceOptions = persistenceOptions;
@@ -234,7 +230,9 @@ export class SessionPool extends EventEmitter {
return;
}
- this.keyValueStore = await KeyValueStore.open(this.persistStateKeyValueStoreId, { config: this.config });
+ this.keyValueStore = await KeyValueStore.open(this.persistStateKeyValueStoreId, {
+ config: serviceLocator.getConfiguration(),
+ });
if (!this.persistenceOptions.enable) {
this.isInitialized = true;
return;
@@ -384,7 +382,7 @@ export class SessionPool extends EventEmitter {
});
// use half the interval of `persistState` to avoid race conditions
- const persistStateIntervalMillis = this.config.get('persistStateIntervalMillis')!;
+ const persistStateIntervalMillis = serviceLocator.getConfiguration().get('persistStateIntervalMillis')!;
const timeoutSecs = persistStateIntervalMillis / 2_000;
await this.keyValueStore
.setValue(this.persistStateKey, this.getState(), {
@@ -528,8 +526,8 @@ export class SessionPool extends EventEmitter {
*
* For more details and code examples, see the {@apilink SessionPool} class.
*/
- static async open(options?: SessionPoolOptions, config?: Configuration): Promise<SessionPool> {
- const sessionPool = new SessionPool(options, config);
+ static async open(options?: SessionPoolOptions): Promise<SessionPool> {
+ const sessionPool = new SessionPool(options);
await sessionPool.initialize();
return sessionPool;
}
diff --git a/packages/core/src/storages/dataset.ts b/packages/core/src/storages/dataset.ts
index 723521c2e..29890faef 100644
--- a/packages/core/src/storages/dataset.ts
+++ b/packages/core/src/storages/dataset.ts
@@ -6,6 +6,7 @@ import { MAX_PAYLOAD_SIZE_BYTES } from '@apify/consts';
import { Configuration } from '../configuration.js';
import { type Log, log } from '../log.js';
+import { serviceLocator } from '../service_locator.js';
import type { Awaitable } from '../typedefs.js';
import { checkStorageAccess } from './access_checking.js';
import { KeyValueStore } from './key_value_store.js';
@@ -686,7 +687,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {
checkStorageAccess();
await this.client.delete();
- const manager = StorageManager.getManager(Dataset, this.config);
+ const manager = StorageManager.getManager(Dataset);
manager.closeStorage(this);
}
@@ -720,11 +721,11 @@ export class Dataset<Data extends Dictionary = Dictionary> {
);
options.config ??= Configuration.getGlobalConfig();
- options.storageClient ??= options.config.getStorageClient();
+ options.storageClient ??= serviceLocator.getStorageClient();
await purgeDefaultStorages({ onlyPurgeOnce: true, client: options.storageClient, config: options.config });
- const manager = StorageManager.getManager<Dataset<Data>>(this, options.config);
+ const manager = StorageManager.getManager<Dataset<Data>>(this);
return manager.openStorage(datasetIdOrName, options.storageClient);
}
diff --git a/packages/core/src/storages/key_value_store.ts b/packages/core/src/storages/key_value_store.ts
index 1752eef23..388571b7d 100644
--- a/packages/core/src/storages/key_value_store.ts
+++ b/packages/core/src/storages/key_value_store.ts
@@ -10,6 +10,7 @@ import log from '@apify/log';
import { jsonStringifyExtended } from '@apify/utilities';
import { Configuration } from '../configuration.js';
+import { serviceLocator } from '../service_locator.js';
import type { Awaitable } from '../typedefs.js';
import { checkStorageAccess } from './access_checking.js';
import type { StorageManagerOptions } from './storage_manager.js';
@@ -278,7 +279,7 @@ export class KeyValueStore {
const persistStateIntervalMillis = this.config.get('persistStateIntervalMillis')!;
const timeoutSecs = persistStateIntervalMillis / 2_000;
- this.config.getEventManager().on('persistState', async () => {
+ serviceLocator.getEventManager().on('persistState', async () => {
const promises: Promise<void>[] = [];
for (const [key, value] of this.cache) {
@@ -417,7 +418,7 @@ export class KeyValueStore {
checkStorageAccess();
await this.client.delete();
- const manager = StorageManager.getManager(KeyValueStore, this.config);
+ const manager = StorageManager.getManager(KeyValueStore);
manager.closeStorage(this);
}
@@ -609,11 +610,11 @@ export class KeyValueStore {
);
options.config ??= Configuration.getGlobalConfig();
- options.storageClient ??= options.config.getStorageClient();
+ options.storageClient ??= serviceLocator.getStorageClient();
await purgeDefaultStorages({ onlyPurgeOnce: true, client: options.storageClient, config: options.config });
- const manager = StorageManager.getManager(this, options.config);
+ const manager = StorageManager.getManager(this);
return manager.openStorage(storeIdOrName, options.storageClient);
}
diff --git a/packages/core/src/storages/request_list.ts b/packages/core/src/storages/request_list.ts
index 1ce31de34..58f5f5f7d 100644
--- a/packages/core/src/storages/request_list.ts
+++ b/packages/core/src/storages/request_list.ts
@@ -2,13 +2,13 @@ import type { BaseHttpClient, Dictionary } from '@crawlee/types';
import { downloadListOfUrls } from '@crawlee/utils';
import ow, { ArgumentError } from 'ow';
-import { Configuration } from '../configuration.js';
-import type { EventManager } from '../events/event_manager.js';
+import type { Configuration } from '../configuration.js';
import { EventType } from '../events/event_manager.js';
import { log } from '../log.js';
import type { ProxyConfiguration } from '../proxy_configuration.js';
import { type InternalSource, Request, type RequestOptions, type Source } from '../request.js';
import { createDeserialize, serializeArray } from '../serialization.js';
+import { serviceLocator } from '../service_locator.js';
import { KeyValueStore } from './key_value_store.js';
import { purgeDefaultStorages } from './utils.js';
@@ -354,7 +354,6 @@ export class RequestList implements IRequestList {
private sources: RequestListSource[];
private sourcesFunction?: RequestListSourcesFunction;
private proxyConfiguration?: ProxyConfiguration;
- private events: EventManager;
private httpClient?: BaseHttpClient;
/**
@@ -371,7 +370,6 @@ export class RequestList implements IRequestList {
state,
proxyConfiguration,
keepDuplicateUrls = false,
- config = Configuration.getGlobalConfig(),
httpClient,
} = options;
@@ -402,7 +400,6 @@ export class RequestList implements IRequestList {
this.persistStateKey = persistStateKey ? `SDK_${persistStateKey}` : persistStateKey;
this.persistRequestsKey = persistRequestsKey ? `SDK_${persistRequestsKey}` : persistRequestsKey;
this.initialState = state;
- this.events = config.getEventManager();
this.httpClient = httpClient;
// If this option is set then all requests will get a pre-generated unique ID and duplicate URLs will be kept in the list.
@@ -444,7 +441,7 @@ export class RequestList implements IRequestList {
this.isInitialized = true;
if (this.persistRequestsKey && !this.areRequestsPersisted) await this._persistRequests();
if (this.persistStateKey) {
- this.events.on(EventType.PERSIST_STATE, this.persistState);
+ serviceLocator.getEventManager().on(EventType.PERSIST_STATE, this.persistState);
}
return this;
@@ -538,7 +535,7 @@ export class RequestList implements IRequestList {
* leaking the listener (and the requests it retains) on the shared event manager.
*/
async teardown(): Promise<void> {
- this.events.off(EventType.PERSIST_STATE, this.persistState);
+ serviceLocator.getEventManager().off(EventType.PERSIST_STATE, this.persistState);
if (this.persistStateKey) {
await this.persistState();
diff --git a/packages/core/src/storages/request_provider.ts b/packages/core/src/storages/request_provider.ts
index 385b7631c..1edbe7e46 100644
--- a/packages/core/src/storages/request_provider.ts
+++ b/packages/core/src/storages/request_provider.ts
@@ -27,11 +27,13 @@ import type { Log } from '@apify/log';
import { cryptoRandomObjectId } from '@apify/utilities';
import { Configuration } from '../configuration.js';
+import type { EventManager } from '../events/event_manager.js';
import { EventType } from '../events/event_manager.js';
import { log } from '../log.js';
import type { ProxyConfiguration } from '../proxy_configuration.js';
import type { InternalSource, RequestOptions, Source } from '../request.js';
import { Request } from '../request.js';
+import { serviceLocator } from '../service_locator.js';
import type { Constructor } from '../typedefs.js';
import { checkStorageAccess } from './access_checking.js';
import { RequestDeduplicationCache } from './request_dedup_cache.js';
@@ -150,12 +152,15 @@ export abstract class RequestProvider implements IStorage, IRequestManager {
protected httpClient?: BaseHttpClient;
+ protected readonly events: EventManager;
+
constructor(
options: InternalRequestProviderOptions,
- readonly config = Configuration.getGlobalConfig(),
+ protected readonly config: Configuration,
) {
this.id = options.id;
this.name = options.name;
+ this.events = serviceLocator.getEventManager();
this.client = options.client.requestQueue(this.id, {
clientKey: this.clientKey,
timeoutSecs: this.timeoutSecs,
@@ -168,9 +173,7 @@ export abstract class RequestProvider implements IStorage, IRequestManager {
this.recentlyHandledRequestsCache = new LruCache({ maxLength: options.recentlyHandledRequestsMaxSize });
this.log = log.child({ prefix: `${options.logPrefix}(${this.id}, ${this.name ?? 'no-name'})` });
- const eventManager = config.getEventManager();
-
- eventManager.on(EventType.MIGRATING, async () => {
+ this.events.on(EventType.MIGRATING, async () => {
this.queuePausedForMigration = true;
});
}
@@ -804,7 +807,7 @@ export abstract class RequestProvider implements IStorage, IRequestManager {
checkStorageAccess();
await this.client.delete();
- const manager = StorageManager.getManager(this.constructor as Constructor<IStorage>, this.config);
+ const manager = StorageManager.getManager(this.constructor as Constructor<IStorage>);
manager.closeStorage(this);
}
@@ -959,12 +962,11 @@ export abstract class RequestProvider implements IStorage, IRequestManager {
}),
);
- options.config ??= Configuration.getGlobalConfig();
- options.storageClient ??= options.config.getStorageClient();
+ options.storageClient ??= serviceLocator.getStorageClient();
await purgeDefaultStorages({ onlyPurgeOnce: true, client: options.storageClient, config: options.config });
- const manager = StorageManager.getManager(this as typeof BuiltRequestProvider, options.config);
+ const manager = StorageManager.getManager(this as typeof BuiltRequestProvider);
const queue = await manager.openStorage(queueIdOrName, options.storageClient);
queue.proxyConfiguration = options.proxyConfiguration;
diff --git a/packages/core/src/storages/request_queue.ts b/packages/core/src/storages/request_queue.ts
index 84c80767f..989301e35 100644
--- a/packages/core/src/storages/request_queue.ts
+++ b/packages/core/src/storages/request_queue.ts
@@ -4,8 +4,9 @@ import type { Dictionary } from '@crawlee/types';
import { REQUEST_QUEUE_HEAD_MAX_LIMIT } from '@apify/consts';
-import { Configuration } from '../configuration.js';
+import type { Configuration } from '../configuration.js';
import type { Request } from '../request.js';
+import { serviceLocator } from '../service_locator.js';
import { checkStorageAccess } from './access_checking.js';
import type { RequestProviderOptions, RequestQueueOperationInfo } from './request_provider.js';
import { RequestProvider } from './request_provider.js';
@@ -87,7 +88,7 @@ class RequestQueue extends RequestProvider {
/**
* @internal
*/
- constructor(options: RequestProviderOptions, config = Configuration.getGlobalConfig()) {
+ constructor(options: RequestProviderOptions, config: Configuration = serviceLocator.getConfiguration()) {
super(
{
...options,
diff --git a/packages/core/src/storages/request_queue_v2.ts b/packages/core/src/storages/request_queue_v2.ts
index 106952a1b..dcacd16bc 100644
--- a/packages/core/src/storages/request_queue_v2.ts
+++ b/packages/core/src/storages/request_queue_v2.ts
@@ -1,8 +1,9 @@
import type { BatchAddRequestsResult, Dictionary } from '@crawlee/types';
-import { Configuration } from '../configuration.js';
+import type { Configuration } from '../configuration.js';
import { EventType } from '../events/event_manager.js';
import type { Request, Source } from '../request.js';
+import { serviceLocator } from '../service_locator.js';
import { checkStorageAccess } from './access_checking.js';
import type {
RequestProviderOptions,
@@ -67,7 +68,7 @@ export class RequestQueue extends RequestProvider {
private shouldCheckForForefrontRequests = false;
private dequeuedRequestCount = 0;
- constructor(options: RequestProviderOptions, config = Configuration.getGlobalConfig()) {
+ constructor(options: RequestProviderOptions, config: Configuration = serviceLocator.getConfiguration()) {
super(
{
...options,
@@ -78,13 +79,11 @@ export class RequestQueue extends RequestProvider {
config,
);
- const eventManager = config.getEventManager();
-
- eventManager.on(EventType.MIGRATING, async () => {
+ this.events.on(EventType.MIGRATING, async () => {
await this._clearPossibleLocks();
});
- eventManager.on(EventType.ABORTING, async () => {
+ this.events.on(EventType.ABORTING, async () => {
await this._clearPossibleLocks();
});
}
diff --git a/packages/core/src/storages/sitemap_request_list.ts b/packages/core/src/storages/sitemap_request_list.ts
index 8c0c950e7..1f0039d11 100644
--- a/packages/core/src/storages/sitemap_request_list.ts
+++ b/packages/core/src/storages/sitemap_request_list.ts
@@ -8,11 +8,11 @@ import type { RequiredDeep } from 'type-fest';
import defaultLog from '@apify/log';
-import { Configuration } from '../configuration.js';
import type { GlobInput, RegExpInput, UrlPatternObject } from '../enqueue_links/shared.js';
import { constructGlobObjectsFromGlobs, constructRegExpObjectsFromRegExps } from '../enqueue_links/shared.js';
import { type EventManager, EventType } from '../events/event_manager.js';
import { Request } from '../request.js';
+import { serviceLocator } from '../service_locator.js';
import { KeyValueStore } from './key_value_store.js';
import type { IRequestList } from './request_list.js';
import { purgeDefaultStorages } from './utils.js';
@@ -107,10 +107,6 @@ export interface SitemapRequestListOptions extends UrlConstraints {
* Advanced options for the underlying `parseSitemap` call.
*/
parseSitemapOptions?: Omit<ParseSitemapOptions, 'emitNestedSitemaps' | 'maxDepth'>;
- /**
- * Crawlee configuration
- */
- config?: Configuration;
/**
* Custom HTTP client to be used for sitemap loading.
*/
@@ -245,7 +241,7 @@ export class SitemapRequestList implements IRequestList {
}),
);
- const { globs, exclude, regexps, config = Configuration.getGlobalConfig() } = options;
+ const { globs, exclude, regexps } = options;
if (exclude?.length) {
for (const excl of exclude) {
@@ -274,7 +270,7 @@ export class SitemapRequestList implements IRequestList {
this.urlQueueStream = this.createNewStream(options.maxBufferSize ?? 200);
this.sitemapParsingProgress.pendingSitemapUrls = new Set(options.sitemapUrls);
- this.events = config.getEventManager();
+ this.events = serviceLocator.getEventManager();
this.persistState = this.persistState.bind(this);
}
diff --git a/packages/core/src/storages/storage_manager.ts b/packages/core/src/storages/storage_manager.ts
index c50ee160f..634132a32 100644
--- a/packages/core/src/storages/storage_manager.ts
+++ b/packages/core/src/storages/storage_manager.ts
@@ -1,8 +1,9 @@
-import type { BaseHttpClient, Dictionary, StorageClient } from '@crawlee/types';
+import type { BaseHttpClient, StorageClient } from '@crawlee/types';
import { AsyncQueue } from '@sapphire/async-queue';
-import { Configuration } from '../configuration.js';
+import type { Configuration } from '../configuration.js';
import type { ProxyConfiguration } from '../proxy_configuration.js';
+import { serviceLocator } from '../service_locator.js';
import type { Constructor } from '../typedefs.js';
const DEFAULT_ID_CONFIG_KEYS = {
@@ -28,7 +29,7 @@ export class StorageManager<T extends IStorage = IStorage> {
constructor(
StorageConstructor: Constructor<T>,
- private readonly config = Configuration.getGlobalConfig(),
+ private readonly config: Configuration,
) {
this.StorageConstructor = StorageConstructor;
this.name = this.StorageConstructor.name as 'Dataset' | 'KeyValueStore' | 'RequestQueue';
@@ -38,33 +39,18 @@ export class StorageManager<T extends IStorage = IStorage> {
storageClass: Constructor<T>,
idOrName?: string,
client?: StorageClient,
- config = Configuration.getGlobalConfig(),
): Promise<T> {
- return this.getManager(storageClass, config).openStorage(idOrName, client);
+ return this.getManager(storageClass).openStorage(idOrName, client);
}
- static getManager<T extends IStorage>(
- storageClass: Constructor<T>,
- config = Configuration.getGlobalConfig(),
- ): StorageManager<T> {
- if (!config.storageManagers.has(storageClass)) {
- const manager = new StorageManager(storageClass, config);
- config.storageManagers.set(storageClass, manager);
+ static getManager<T extends IStorage>(storageClass: Constructor<T>): StorageManager<T> {
+ let storageManager = serviceLocator.getStorageManager(storageClass);
+ if (storageManager === undefined) {
+ storageManager = new StorageManager(storageClass, serviceLocator.getConfiguration());
+ serviceLocator.setStorageManager(storageClass, storageManager);
}
- return config.storageManagers.get(storageClass) as StorageManager<T>;
- }
-
- /** @internal */
- static clearCache(config = Configuration.getGlobalConfig()): void {
- config.storageManagers.forEach((manager) => {
- if (manager.name === 'KeyValueStore') {
- manager.cache.forEach((item) => {
- (item as Dictionary).clearCache?.();
- });
- }
- });
- config.storageManagers.clear();
+ return serviceLocator.getStorageManager(storageClass) as StorageManager<T>;
}
async openStorage(idOrName?: string | null, client?: StorageClient): Promise<T> {
@@ -80,7 +66,7 @@ export class StorageManager<T extends IStorage = IStorage> {
let storage = this.cache.get(cacheKey);
if (!storage) {
- client ??= this.config.getStorageClient();
+ client ??= serviceLocator.getStorageClient();
const storageObject = await this._getOrCreateStorage(idOrName, this.name, client);
storage = new this.StorageConstructor(
{
diff --git a/packages/core/src/storages/utils.ts b/packages/core/src/storages/utils.ts
index dc0450e85..ec0881162 100644
--- a/packages/core/src/storages/utils.ts
+++ b/packages/core/src/storages/utils.ts
@@ -3,6 +3,7 @@ import crypto from 'node:crypto';
import type { Dictionary, StorageClient } from '@crawlee/types';
import { Configuration } from '../configuration.js';
+import { serviceLocator } from '../service_locator.js';
import { KeyValueStore } from './key_value_store.js';
/**
@@ -54,8 +55,8 @@ export async function purgeDefaultStorages(
config: configOrOptions,
}
: (configOrOptions ?? {});
- const { config = Configuration.getGlobalConfig(), onlyPurgeOnce = false } = options;
- ({ client = config.getStorageClient() } = options);
+ const { config = serviceLocator.getConfiguration(), onlyPurgeOnce = false } = options;
+ ({ client = serviceLocator.getStorageClient() } = options);
const casted = client as StorageClient & { __purged?: Promise<void> };
@@ -100,7 +101,7 @@ export async function useState<State extends Dictionary = Dictionary>(
options?: UseStateOptions,
) {
const kvStore = await KeyValueStore.open(options?.keyValueStoreName, {
- config: options?.config || Configuration.getGlobalConfig(),
+ config: options?.config || serviceLocator.getConfiguration(),
});
return kvStore.getAutoSavedValue<State>(name || 'CRAWLEE_GLOBAL_STATE', defaultValue);
}
diff --git a/packages/core/test/core/service_locator.test.ts b/packages/core/test/core/service_locator.test.ts
new file mode 100644
index 000000000..8c4c6afb8
--- /dev/null
+++ b/packages/core/test/core/service_locator.test.ts
@@ -0,0 +1,229 @@
+import { Configuration, LocalEventManager, ServiceConflictError, ServiceLocator, serviceLocator } from '@crawlee/core';
+import { MemoryStorage } from '@crawlee/memory-storage';
+
+// Reset global service locator before each test
+beforeEach(() => {
+ serviceLocator.reset();
+});
+
+describe('ServiceLocator', () => {
+ describe('Configuration', () => {
+ test('default configuration', () => {
+ const config = serviceLocator.getConfiguration();
+
+ // Should return a Configuration instance
+ expect(config).toBeInstanceOf(Configuration);
+ });
+
+ test('custom configuration', () => {
+ const customConfig = new Configuration({ headless: false });
+ serviceLocator.setConfiguration(customConfig);
+ const config = serviceLocator.getConfiguration();
+
+ expect(config).toBe(customConfig);
+ });
+
+ test('configuration overwrite not possible', () => {
+ const defaultConfig = new Configuration();
+ serviceLocator.setConfiguration(defaultConfig);
+
+ const customConfig = new Configuration({ headless: false });
+
+ expect(() => {
+ serviceLocator.setConfiguration(customConfig);
+ }).toThrow(ServiceConflictError);
+ });
+
+ test('configuration conflict', () => {
+ // Retrieve configuration first
+ serviceLocator.getConfiguration();
+
+ const customConfig = new Configuration({ headless: false });
+
+ expect(() => {
+ serviceLocator.setConfiguration(customConfig);
+ }).toThrow(ServiceConflictError);
+ expect(() => {
+ serviceLocator.setConfiguration(customConfig);
+ }).toThrow(/Configuration is already in use/);
+ });
+ });
+
+ describe('EventManager', () => {
+ test('default event manager', () => {
+ const defaultEventManager = serviceLocator.getEventManager();
+ expect(defaultEventManager).toBeInstanceOf(LocalEventManager);
+ });
+
+ test('custom event manager', () => {
+ const customEventManager = new LocalEventManager({
+ persistStateIntervalMillis: 1000,
+ systemInfoIntervalMillis: 1000,
+ });
+ serviceLocator.setEventManager(customEventManager);
+ const eventManager = serviceLocator.getEventManager();
+
+ expect(eventManager).toBe(customEventManager);
+ });
+
+ test('event manager overwrite not possible', () => {
+ const customEventManager = new LocalEventManager({
+ persistStateIntervalMillis: 1000,
+ systemInfoIntervalMillis: 1000,
+ });
+ serviceLocator.setEventManager(customEventManager);
+
+ const anotherCustomEventManager = new LocalEventManager({
+ persistStateIntervalMillis: 1000,
+ systemInfoIntervalMillis: 1000,
+ });
+
+ expect(() => {
+ serviceLocator.setEventManager(anotherCustomEventManager);
+ }).toThrow(ServiceConflictError);
+ });
+
+ test('event manager conflict', () => {
+ // Retrieve event manager first
+ serviceLocator.getEventManager();
+
+ const customEventManager = new LocalEventManager({
+ persistStateIntervalMillis: 1000,
+ systemInfoIntervalMillis: 1000,
+ });
+
+ expect(() => {
+ serviceLocator.setEventManager(customEventManager);
+ }).toThrow(ServiceConflictError);
+ expect(() => {
+ serviceLocator.setEventManager(customEventManager);
+ }).toThrow(/EventManager is already in use/);
+ });
+ });
+
+ describe('StorageClient', () => {
+ test('default storage client', () => {
+ const defaultStorageClient = serviceLocator.getStorageClient();
+ expect(defaultStorageClient).toBeInstanceOf(MemoryStorage);
+ });
+
+ test('custom storage client', () => {
+ const customStorageClient = new MemoryStorage();
+ serviceLocator.setStorageClient(customStorageClient);
+ const storageClient = serviceLocator.getStorageClient();
+
+ expect(storageClient).toBe(customStorageClient);
+ });
+
+ test('storage client overwrite not possible', () => {
+ const customStorageClient = new MemoryStorage();
+ serviceLocator.setStorageClient(customStorageClient);
+
+ const anotherCustomStorageClient = new MemoryStorage();
+
+ expect(() => {
+ serviceLocator.setStorageClient(anotherCustomStorageClient);
+ }).toThrow(ServiceConflictError);
+ });
+
+ test('storage client conflict', () => {
+ // Retrieve storage client first
+ serviceLocator.getStorageClient();
+
+ const customStorageClient = new MemoryStorage();
+
+ expect(() => {
+ serviceLocator.setStorageClient(customStorageClient);
+ }).toThrow(ServiceConflictError);
+ expect(() => {
+ serviceLocator.setStorageClient(customStorageClient);
+ }).toThrow(/StorageClient is already in use/);
+ });
+ });
+
+ describe('Reset functionality', () => {
+ test('reset clears all services', () => {
+ const customConfig = new Configuration({ headless: false });
+ const customEventManager = new LocalEventManager({
+ persistStateIntervalMillis: 1000,
+ systemInfoIntervalMillis: 1000,
+ });
+ const customStorageClient = new MemoryStorage();
+
+ serviceLocator.setConfiguration(customConfig);
+ serviceLocator.setEventManager(customEventManager);
+ serviceLocator.setStorageClient(customStorageClient);
+
+ // Verify they're set
+ expect(serviceLocator.getConfiguration()).toBe(customConfig);
+ expect(serviceLocator.getEventManager()).toBe(customEventManager);
+ expect(serviceLocator.getStorageClient()).toBe(customStorageClient);
+
+ // Reset
+ serviceLocator.reset();
+
+ // After reset, should be able to set new instances
+ const newConfig = new Configuration({ headless: true });
+ serviceLocator.setConfiguration(newConfig);
+ expect(serviceLocator.getConfiguration()).toBe(newConfig);
+ });
+ });
+
+ describe('Same instance allowed', () => {
+ test('setting same configuration instance is allowed', () => {
+ const config = new Configuration();
+ serviceLocator.setConfiguration(config);
+ serviceLocator.getConfiguration();
+
+ // Setting the same instance again should not throw
+ expect(() => {
+ serviceLocator.setConfiguration(config);
+ }).not.toThrow();
+ });
+
+ test('setting same event manager instance is allowed', () => {
+ const eventManager = new LocalEventManager({
+ persistStateIntervalMillis: 1000,
+ systemInfoIntervalMillis: 1000,
+ });
+ serviceLocator.setEventManager(eventManager);
+ serviceLocator.getEventManager();
+
+ // Setting the same instance again should not throw
+ expect(() => {
+ serviceLocator.setEventManager(eventManager);
+ }).not.toThrow();
+ });
+
+ test('setting same storage client instance is allowed', () => {
+ const storageClient = new MemoryStorage();
+ serviceLocator.setStorageClient(storageClient);
+ serviceLocator.getStorageClient();
+
+ // Setting the same instance again should not throw
+ expect(() => {
+ serviceLocator.setStorageClient(storageClient);
+ }).not.toThrow();
+ });
+ });
+
+ describe('Per-crawler ServiceLocator', () => {
+ test('creating separate service locator for crawler', () => {
+ const crawlerConfig = new Configuration({ headless: false });
+ const crawlerStorage = new MemoryStorage();
+ const crawlerEvents = new LocalEventManager({
+ persistStateIntervalMillis: 1000,
+ systemInfoIntervalMillis: 1000,
+ });
+
+ const crawlerLocator = new ServiceLocator(crawlerConfig, crawlerEvents, crawlerStorage);
+
+ expect(crawlerLocator.getConfiguration()).toBe(crawlerConfig);
+ expect(crawlerLocator.getEventManager()).toBe(crawlerEvents);
+ expect(crawlerLocator.getStorageClient()).toBe(crawlerStorage);
+
+ // Global service locator should remain independent
+ expect(serviceLocator.getConfiguration()).not.toBe(crawlerConfig);
+ });
+ });
+});
diff --git a/packages/core/test/enqueue_links/protocol-matching-based-on-strategy.test.ts b/packages/core/test/enqueue_links/protocol-matching-based-on-strategy.test.ts
index 139aba7d8..576f86587 100644
--- a/packages/core/test/enqueue_links/protocol-matching-based-on-strategy.test.ts
+++ b/packages/core/test/enqueue_links/protocol-matching-based-on-strategy.test.ts
@@ -1,10 +1,10 @@
import { load } from 'cheerio';
import type { CheerioRoot, Source } from 'crawlee';
-import { cheerioCrawlerEnqueueLinks, Configuration, EnqueueStrategy, RequestQueue } from 'crawlee';
+import { cheerioCrawlerEnqueueLinks, EnqueueStrategy, RequestQueue, serviceLocator } from 'crawlee';
import log from '@apify/log';
-const apifyClient = Configuration.getStorageClient();
+const apifyClient = serviceLocator.getStorageClient();
const HTML = `
<html>
@@ -22,7 +22,7 @@ const HTML = `
function createRequestQueueMock() {
const enqueued: Source[] = [];
- const requestQueue = new RequestQueue({ id: 'xxx', client: apifyClient });
+ const requestQueue = new RequestQueue({ id: 'xxx', client: apifyClient }, serviceLocator.getConfiguration());
// @ts-expect-error Override method for testing
requestQueue.addRequests = async function (requests) {
diff --git a/packages/core/test/enqueue_links/user-provided-patterns-with-enqueue-strategy.test.ts b/packages/core/test/enqueue_links/user-provided-patterns-with-enqueue-strategy.test.ts
index d3ee7a163..bac01890f 100644
--- a/packages/core/test/enqueue_links/user-provided-patterns-with-enqueue-strategy.test.ts
+++ b/packages/core/test/enqueue_links/user-provided-patterns-with-enqueue-strategy.test.ts
@@ -1,10 +1,10 @@
import { load } from 'cheerio';
import type { CheerioRoot, Source } from 'crawlee';
-import { cheerioCrawlerEnqueueLinks, Configuration, EnqueueStrategy, RequestQueue } from 'crawlee';
+import { cheerioCrawlerEnqueueLinks, EnqueueStrategy, RequestQueue, serviceLocator } from 'crawlee';
import log from '@apify/log';
-const apifyClient = Configuration.getStorageClient();
+const apifyClient = serviceLocator.getStorageClient();
const HTML = `
<html>
@@ -34,7 +34,7 @@ const HTML = `
function createRequestQueueMock() {
const enqueued: Source[] = [];
- const requestQueue = new RequestQueue({ id: 'xxx', client: apifyClient });
+ const requestQueue = new RequestQueue({ id: 'xxx', client: apifyClient }, serviceLocator.getConfiguration());
// @ts-expect-error Override method for testing
requestQueue.addRequests = async function (requests) {
diff --git a/packages/core/test/enqueue_links/userData.test.ts b/packages/core/test/enqueue_links/userData.test.ts
index 82bb59c8e..74f43d036 100644
--- a/packages/core/test/enqueue_links/userData.test.ts
+++ b/packages/core/test/enqueue_links/userData.test.ts
@@ -1,11 +1,11 @@
import type { Source } from '@crawlee/cheerio';
-import { cheerioCrawlerEnqueueLinks, Configuration, RequestQueue } from '@crawlee/cheerio';
+import { cheerioCrawlerEnqueueLinks, RequestQueue, serviceLocator } from '@crawlee/cheerio';
import type { CheerioAPI } from 'cheerio';
import { load } from 'cheerio';
import log from '@apify/log';
-const apifyClient = Configuration.getStorageClient();
+const apifyClient = serviceLocator.getStorageClient();
const HTML = `
<html>
@@ -23,7 +23,7 @@ const HTML = `
function createRequestQueueMock() {
const enqueued: Source[] = [];
- const requestQueue = new RequestQueue({ id: 'xxx', client: apifyClient });
+ const requestQueue = new RequestQueue({ id: 'xxx', client: apifyClient }, serviceLocator.getConfiguration());
// @ts-expect-error Override method for testing
requestQueue.addRequests = async function (requests) {
diff --git a/packages/core/test/request-queue/adding-the-same-request-should-not-call-the-api.test.ts b/packages/core/test/request-queue/adding-the-same-request-should-not-call-the-api.test.ts
index d5c17388d..cd212bbc5 100644
--- a/packages/core/test/request-queue/adding-the-same-request-should-not-call-the-api.test.ts
+++ b/packages/core/test/request-queue/adding-the-same-request-should-not-call-the-api.test.ts
@@ -1,25 +1,22 @@
import { MemoryStorage } from '@crawlee/memory-storage';
import type { RequestQueueInfo } from '@crawlee/types';
-import { Configuration, RequestQueue } from 'crawlee';
-
-const originalClient = Configuration.getStorageClient();
-Configuration.useStorageClient(new MemoryStorage({ persistStorage: false, writeMetadata: false }));
-
-afterAll(() => {
- Configuration.useStorageClient(originalClient);
-});
+import { RequestQueue, serviceLocator } from 'crawlee';
let requestQueueInfo: RequestQueueInfo;
-beforeAll(async () => {
- requestQueueInfo = await Configuration.getStorageClient()
+beforeEach(async () => {
+ serviceLocator.setStorageClient(new MemoryStorage({ persistStorage: false, writeMetadata: false }));
+ requestQueueInfo = await serviceLocator
+ .getStorageClient()
.requestQueues()
.getOrCreate('test-request-queue-not-called-on-cached-request');
});
describe('RequestQueue#addRequest should not call the API if the request is already in the queue', () => {
test('should not call the API if the request is already in the queue', async () => {
- const requestQueue = new RequestQueue({ id: requestQueueInfo.id, client: Configuration.getStorageClient() });
+ const client = serviceLocator.getStorageClient();
+ const config = serviceLocator.getConfiguration();
+ const requestQueue = new RequestQueue({ id: requestQueueInfo.id, client }, config);
const clientSpy = vitest.spyOn(requestQueue.client, 'addRequest');
@@ -41,7 +38,9 @@ describe('RequestQueue#addRequest should not call the API if the request is alre
describe('RequestQueue#addRequests should not call the API if the request is already in the queue', () => {
test('should not call the API if the request is already in the queue', async () => {
- const requestQueue = new RequestQueue({ id: requestQueueInfo.id, client: Configuration.getStorageClient() });
+ const client = serviceLocator.getStorageClient();
+ const config = serviceLocator.getConfiguration();
+ const requestQueue = new RequestQueue({ id: requestQueueInfo.id, client }, config);
const clientSpy = vitest.spyOn(requestQueue.client, 'batchAddRequests');
diff --git a/packages/core/test/storages/open-storage-with-different-client-should-be-respected.test.ts b/packages/core/test/storages/open-storage-with-different-client-should-be-respected.test.ts
index 36281829e..48b15ed25 100644
--- a/packages/core/test/storages/open-storage-with-different-client-should-be-respected.test.ts
+++ b/packages/core/test/storages/open-storage-with-different-client-should-be-respected.test.ts
@@ -1,12 +1,11 @@
import { MemoryStorage } from '@crawlee/memory-storage';
-import { Configuration, RequestQueue } from 'crawlee';
+import { RequestQueue, serviceLocator } from 'crawlee';
-const originalClient = Configuration.getStorageClient();
-const newClient = new MemoryStorage({ persistStorage: false, writeMetadata: false });
-Configuration.useStorageClient(newClient);
+let newClient: MemoryStorage;
-afterAll(() => {
- Configuration.useStorageClient(originalClient);
+beforeEach(() => {
+ newClient = new MemoryStorage({ persistStorage: false, writeMetadata: false });
+ serviceLocator.setStorageClient(newClient);
});
describe('Opening a storage with a different storage client should be respected', () => {
diff --git a/packages/http-crawler/src/internals/http-crawler.ts b/packages/http-crawler/src/internals/http-crawler.ts
index b13b04403..e6cc3be49 100644
--- a/packages/http-crawler/src/internals/http-crawler.ts
+++ b/packages/http-crawler/src/internals/http-crawler.ts
@@ -19,7 +19,6 @@ import type {
import {
BasicCrawler,
BLOCKED_STATUS_CODES,
- Configuration,
ContextPipeline,
mergeCookies,
RequestState,
@@ -353,7 +352,6 @@ export class HttpCrawler<
constructor(
options: HttpCrawlerOptions<Context, ContextExtension, ExtendedContext> &
RequireContextPipeline<InternalHttpCrawlingContext, Context> = {} as any,
- override readonly config = Configuration.getGlobalConfig(),
) {
ow(options, 'HttpCrawlerOptions', ow.object.exactShape(HttpCrawler.optionsShape));
@@ -375,16 +373,13 @@ export class HttpCrawler<
...basicCrawlerOptions
} = options;
- super(
- {
- ...basicCrawlerOptions,
- autoscaledPoolOptions,
- contextPipelineBuilder:
- contextPipelineBuilder ??
- (() => this.buildContextPipeline() as ContextPipeline<CrawlingContext, Context>),
- },
- config,
- );
+ super({
+ ...basicCrawlerOptions,
+ autoscaledPoolOptions,
+ contextPipelineBuilder:
+ contextPipelineBuilder ??
+ (() => this.buildContextPipeline() as ContextPipeline<CrawlingContext, Context>),
+ });
// Cookies should be persisted per session only if session pool is used
if (!this.useSessionPool && persistCookiesPerSession) {
diff --git a/packages/jsdom-crawler/src/internals/jsdom-crawler.ts b/packages/jsdom-crawler/src/internals/jsdom-crawler.ts
index 4bf2a049e..2b9cef56f 100644
--- a/packages/jsdom-crawler/src/internals/jsdom-crawler.ts
+++ b/packages/jsdom-crawler/src/internals/jsdom-crawler.ts
@@ -1,6 +1,5 @@
import type {
BasicCrawlingContext,
- Configuration,
EnqueueLinksOptions,
ErrorHandler,
GetUserDataFromRequest,
@@ -195,25 +194,22 @@ export class JSDOMCrawler<
protected hideInternalConsole: boolean;
protected virtualConsole: VirtualConsole | null = null;
- constructor(options: JSDOMCrawlerOptions<ContextExtension, ExtendedContext> = {}, config?: Configuration) {
+ constructor(options: JSDOMCrawlerOptions<ContextExtension, ExtendedContext> = {}) {
const { runScripts = false, hideInternalConsole = false, ...httpOptions } = options;
- super(
- {
- ...httpOptions,
- contextPipelineBuilder: () =>
- this.buildContextPipeline()
- .compose({
- action: async (context) => await this.parseContent(context),
- cleanup: async (context) => {
- this.getVirtualConsole().off('jsdomError', this.jsdomErrorHandler);
- context.window?.close();
- },
- })
- .compose({ action: async (context) => await this.addHelpers(context) }),
- },
- config,
- );
+ super({
+ ...httpOptions,
+ contextPipelineBuilder: () =>
+ this.buildContextPipeline()
+ .compose({
+ action: async (context) => await this.parseContent(context),
+ cleanup: async (context) => {
+ this.getVirtualConsole().off('jsdomError', this.jsdomErrorHandler);
+ context.window?.close();
+ },
+ })
+ .compose({ action: async (context) => await this.addHelpers(context) }),
+ });
this.runScripts = runScripts;
this.hideInternalConsole = hideInternalConsole;
diff --git a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts
index 64dab6e52..be4a5bea4 100644
--- a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts
+++ b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts
@@ -28,11 +28,11 @@ import type {
StatisticState,
} from '@crawlee/core';
import {
- Configuration,
RequestHandlerError,
RequestHandlerResult,
resolveBaseUrlForEnqueueLinksFiltering,
Router,
+ serviceLocator,
Statistics,
withCheckedStorageAccess,
} from '@crawlee/core';
@@ -310,10 +310,7 @@ export class AdaptivePlaywrightCrawler<
return this.adaptiveRequestHandler as unknown as RequestHandler<PlaywrightCrawlingContext>;
}
- constructor(
- options: AdaptivePlaywrightCrawlerOptions<ExtendedContext> = {},
- override readonly config = Configuration.getGlobalConfig(),
- ) {
+ constructor(options: AdaptivePlaywrightCrawlerOptions<ExtendedContext> = {}) {
const {
requestHandler,
renderingTypeDetectionRatio = 0.1,
@@ -333,21 +330,19 @@ export class AdaptivePlaywrightCrawler<
...rest
} = options;
- super(
- {
- ...rest,
- // Pass error handlers to the "main" crawler - we only pluck them from `rest` so that they don't go to the sub crawlers
- errorHandler,
- failedRequestHandler,
- // Same for request handler
- requestHandler,
- // The builder intentionally returns null so that it crashes the crawler when it tries to use this instead of one of two the specialized context pipelines
- // (that would be a logical error in this class)
- contextPipelineBuilder: () =>
- null as unknown as ContextPipeline<CrawlingContext, AdaptivePlaywrightCrawlerContext>,
- },
- config,
- );
+ super({
+ ...rest,
+ // Pass error handlers to the "main" crawler - we only pluck them from `rest` so that they don't go to the sub crawlers
+ errorHandler,
+ failedRequestHandler,
+ // Same for request handler
+ requestHandler,
+ // The builder intentionally returns null so that it crashes the crawler when it tries to use this instead of one of two the specialized context pipelines
+ // (that would be a logical error in this class)
+ contextPipelineBuilder: () =>
+ null as unknown as ContextPipeline<CrawlingContext, AdaptivePlaywrightCrawlerContext>,
+ });
+
this.individualRequestHandlerTimeoutMillis = requestHandlerTimeoutSecs * 1000;
this.renderingTypePredictor =
@@ -370,55 +365,49 @@ export class AdaptivePlaywrightCrawler<
);
};
}
- const staticCrawler = new CheerioCrawler(
- {
- ...rest,
- useSessionPool: false,
- statisticsOptions: {
- persistenceOptions: { enable: false },
- },
- preNavigationHooks: [
- async (context) => {
- for (const hook of preNavigationHooks ?? []) {
- await hook(context, undefined);
- }
- },
- ],
- postNavigationHooks: [
- async (context) => {
- for (const hook of postNavigationHooks ?? []) {
- await hook(context, undefined);
- }
- },
- ],
+ const staticCrawler = new CheerioCrawler({
+ ...rest,
+ useSessionPool: false,
+ statisticsOptions: {
+ persistenceOptions: { enable: false },
},
- config,
- );
-
- const browserCrawler = new PlaywrightCrawler(
- {
- ...rest,
- useSessionPool: false,
- statisticsOptions: {
- persistenceOptions: { enable: false },
+ preNavigationHooks: [
+ async (context) => {
+ for (const hook of preNavigationHooks ?? []) {
+ await hook(context, undefined);
+ }
},
- preNavigationHooks: [
- async (context, gotoOptions) => {
- for (const hook of preNavigationHooks ?? []) {
- await hook(context, gotoOptions);
- }
- },
- ],
- postNavigationHooks: [
- async (context, gotoOptions) => {
- for (const hook of postNavigationHooks ?? []) {
- await hook(context, gotoOptions);
- }
- },
- ],
+ ],
+ postNavigationHooks: [
+ async (context) => {
+ for (const hook of postNavigationHooks ?? []) {
+ await hook(context, undefined);
+ }
+ },
+ ],
+ });
+
+ const browserCrawler = new PlaywrightCrawler({
+ ...rest,
+ useSessionPool: false,
+ statisticsOptions: {
+ persistenceOptions: { enable: false },
},
- config,
- );
+ preNavigationHooks: [
+ async (context, gotoOptions) => {
+ for (const hook of preNavigationHooks ?? []) {
+ await hook(context, gotoOptions);
+ }
+ },
+ ],
+ postNavigationHooks: [
+ async (context, gotoOptions) => {
+ for (const hook of postNavigationHooks ?? []) {
+ await hook(context, gotoOptions);
+ }
+ },
+ ],
+ });
this.teardownHooks.push(browserCrawler.teardown.bind(browserCrawler));
@@ -442,7 +431,6 @@ export class AdaptivePlaywrightCrawler<
this.stats = new AdaptivePlaywrightCrawlerStatistics({
logMessage: `${this.log.getOptions().prefix} request statistics:`,
- config,
...statisticsOptions,
});
@@ -540,7 +528,10 @@ export class AdaptivePlaywrightCrawler<
context: CrawlingContext,
useStateFunction: (defaultValue?: Dictionary) => Promise<Dictionary>,
): Promise<Result<RequestHandlerResult>> {
- const result = new RequestHandlerResult(this.config, AdaptivePlaywrightCrawler.CRAWLEE_STATE_KEY);
+ const result = new RequestHandlerResult(
+ serviceLocator.getConfiguration(),
+ AdaptivePlaywrightCrawler.CRAWLEE_STATE_KEY,
+ );
const logs: LogProxyCall[] = [];
const deferredCleanup: (() => Promise<unknown>)[] = [];
diff --git a/packages/playwright-crawler/src/internals/playwright-crawler.ts b/packages/playwright-crawler/src/internals/playwright-crawler.ts
index 1e969e6e4..b51abec7d 100644
--- a/packages/playwright-crawler/src/internals/playwright-crawler.ts
+++ b/packages/playwright-crawler/src/internals/playwright-crawler.ts
@@ -9,7 +9,7 @@ import type {
RouteSchemas,
RoutesFromSchemas,
} from '@crawlee/browser';
-import { BrowserCrawler, Configuration, RequestState, Router } from '@crawlee/browser';
+import { BrowserCrawler, RequestState, Router, serviceLocator } from '@crawlee/browser';
import type { BrowserPoolOptions, PlaywrightController, PlaywrightPlugin } from '@crawlee/browser-pool';
import type { Dictionary } from '@crawlee/types';
import ow from 'ow';
@@ -202,10 +202,7 @@ export class PlaywrightCrawler<
/**
* All `PlaywrightCrawler` parameters are passed via an options object.
*/
- constructor(
- options: PlaywrightCrawlerOptions<ExtendedContext> = {},
- override readonly config = Configuration.getGlobalConfig(),
- ) {
+ constructor(options: PlaywrightCrawlerOptions<ExtendedContext> = {}) {
ow(options, 'PlaywrightCrawlerOptions', ow.object.exactShape(PlaywrightCrawler.optionsShape));
const { launchContext = {}, headless, ...browserCrawlerOptions } = options;
@@ -232,20 +229,17 @@ export class PlaywrightCrawler<
launchContext.launchOptions.headless = headless as boolean;
}
- const playwrightLauncher = new PlaywrightLauncher(launchContext, config);
+ const playwrightLauncher = new PlaywrightLauncher(launchContext, options.configuration);
browserPoolOptions.browserPlugins = [playwrightLauncher.createBrowserPlugin()];
- super(
- {
- ...(browserCrawlerOptions as PlaywrightCrawlerOptions<ExtendedContext>),
- launchContext,
- browserPoolOptions,
- contextPipelineBuilder: () =>
- this.buildContextPipeline().compose({ action: this.enhanceContext.bind(this) }),
- },
- config,
- );
+ super({
+ ...(browserCrawlerOptions as PlaywrightCrawlerOptions<ExtendedContext>),
+ launchContext,
+ browserPoolOptions,
+ contextPipelineBuilder: () =>
+ this.buildContextPipeline().compose({ action: this.enhanceContext.bind(this) }),
+ });
}
protected override async _navigationHandler(
@@ -291,7 +285,7 @@ export class PlaywrightCrawler<
playwrightUtils.infiniteScroll(context.page, options),
listDownloads: async () => downloads,
saveSnapshot: async (options?: SaveSnapshotOptions) =>
- playwrightUtils.saveSnapshot(context.page, { ...options, config: this.config }),
+ playwrightUtils.saveSnapshot(context.page, { ...options, config: serviceLocator.getConfiguration() }),
enqueueLinksByClickingElements: async (
options: Omit<EnqueueLinksByClickingElementsOptions, 'page' | 'requestQueue'>,
) =>
diff --git a/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts b/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts
index 3772f815b..a2cee00b4 100644
--- a/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts
+++ b/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts
@@ -8,8 +8,9 @@ import type {
RouteSchemas,
RoutesFromSchemas,
} from '@crawlee/browser';
-import { BrowserCrawler, Configuration, RequestState, Router } from '@crawlee/browser';
+import { BrowserCrawler, RequestState, Router } from '@crawlee/browser';
import type { BrowserPoolOptions, PuppeteerController, PuppeteerPlugin } from '@crawlee/browser-pool';
+import { serviceLocator } from '@crawlee/core';
import type { Dictionary } from '@crawlee/types';
import ow from 'ow';
// @ts-ignore This only throws when compiled against puppeteer 25+ (ESM only), we only import types, so its alllll gooooood
@@ -173,10 +174,7 @@ export class PuppeteerCrawler<
/**
* All `PuppeteerCrawler` parameters are passed via an options object.
*/
- constructor(
- options: PuppeteerCrawlerOptions<ContextExtension, ExtendedContext> = {},
- override readonly config = Configuration.getGlobalConfig(),
- ) {
+ constructor(options: PuppeteerCrawlerOptions<ContextExtension, ExtendedContext> = {}) {
ow(options, 'PuppeteerCrawlerOptions', ow.object.exactShape(PuppeteerCrawler.optionsShape));
const { launchContext = {}, headless, proxyConfiguration, ...browserCrawlerOptions } = options;
@@ -203,28 +201,25 @@ export class PuppeteerCrawler<
launchContext.launchOptions.headless = headless as boolean;
}
- const puppeteerLauncher = new PuppeteerLauncher(launchContext, config);
+ const puppeteerLauncher = new PuppeteerLauncher(launchContext, options.configuration);
browserPoolOptions.browserPlugins = [puppeteerLauncher.createBrowserPlugin()];
- super(
- {
- ...(browserCrawlerOptions as BrowserCrawlerOptions<
- Page,
- HTTPResponse,
- PuppeteerController,
- PuppeteerCrawlingContext,
- ContextExtension,
- ExtendedContext
- >),
- launchContext,
- proxyConfiguration,
- browserPoolOptions,
- contextPipelineBuilder: () =>
- this.buildContextPipeline().compose({ action: this.enhanceContext.bind(this) }),
- },
- config,
- );
+ super({
+ ...(browserCrawlerOptions as BrowserCrawlerOptions<
+ Page,
+ HTTPResponse,
+ PuppeteerController,
+ PuppeteerCrawlingContext,
+ ContextExtension,
+ ExtendedContext
+ >),
+ launchContext,
+ proxyConfiguration,
+ browserPoolOptions,
+ contextPipelineBuilder: () =>
+ this.buildContextPipeline().compose({ action: this.enhanceContext.bind(this) }),
+ });
}
private async enhanceContext(context: BrowserCrawlingContext<Page, HTTPResponse, PuppeteerController>) {
@@ -271,7 +266,7 @@ export class PuppeteerCrawler<
infiniteScroll: async (options?: InfiniteScrollOptions) =>
puppeteerUtils.infiniteScroll(context.page, options),
saveSnapshot: async (options?: SaveSnapshotOptions) =>
- puppeteerUtils.saveSnapshot(context.page, { ...options, config: this.config }),
+ puppeteerUtils.saveSnapshot(context.page, { ...options, config: serviceLocator.getConfiguration() }),
closeCookieModals: async () => puppeteerUtils.closeCookieModals(context.page),
};
}
diff --git a/test/core/autoscaling/snapshotter.test.ts b/test/core/autoscaling/snapshotter.test.ts
index fb2fade41..7d133a138 100644
--- a/test/core/autoscaling/snapshotter.test.ts
+++ b/test/core/autoscaling/snapshotter.test.ts
@@ -1,6 +1,6 @@
import os from 'node:os';
-import { Configuration, EventType, LocalEventManager, Snapshotter } from '@crawlee/core';
+import { Configuration, EventType, LocalEventManager, serviceLocator, Snapshotter } from '@crawlee/core';
import type { MemoryInfo } from '@crawlee/utils';
import * as utils from '@crawlee/utils';
import { sleep } from '@crawlee/utils';
@@ -22,15 +22,15 @@ describe('Snapshotter', () => {
});
test('should collect snapshots with some values', async () => {
+ serviceLocator.setConfiguration(new Configuration({ systemInfoIntervalMillis: 100 }));
+
// mock client data
- const apifyClient = Configuration.getStorageClient();
+ const apifyClient = serviceLocator.getStorageClient();
const oldStats = apifyClient.stats;
apifyClient.stats = {} as any;
apifyClient.stats!.rateLimitErrors = [0, 0, 0];
-
- const config = new Configuration({ systemInfoIntervalMillis: 100 });
- const snapshotter = new Snapshotter({ config });
- const events = config.getEventManager();
+ const snapshotter = new Snapshotter();
+ const events = serviceLocator.getEventManager();
await events.init();
await snapshotter.start();
@@ -81,13 +81,13 @@ describe('Snapshotter', () => {
});
test('should override default timers', async () => {
- const config = new Configuration({ systemInfoIntervalMillis: 0.1 });
- const snapshotter = new Snapshotter({ config, eventLoopSnapshotIntervalSecs: 0.05 });
- await config.getEventManager().init();
+ serviceLocator.setConfiguration(new Configuration({ systemInfoIntervalMillis: 0.1 }));
+ const snapshotter = new Snapshotter({ eventLoopSnapshotIntervalSecs: 0.05 });
+ await serviceLocator.getEventManager().init();
await snapshotter.start();
await sleep(3 * 1e3);
await snapshotter.stop();
- await config.getEventManager().close();
+ await serviceLocator.getEventManager().close();
const memorySnapshots = snapshotter.getMemorySample();
const eventLoopSnapshots = snapshotter.getEventLoopSample();
const cpuSnapshots = snapshotter.getCpuSample();
@@ -100,7 +100,7 @@ describe('Snapshotter', () => {
test('correctly marks CPU overloaded using Platform event', async () => {
let count = 0;
const emitAndWait = async (delay: number) => {
- Configuration.getEventManager().emit(EventType.SYSTEM_INFO, {
+ serviceLocator.getEventManager().emit(EventType.SYSTEM_INFO, {
isCpuOverloaded: count % 2 === 0,
createdAt: new Date().toISOString(),
cpuCurrentUsage: 66.6,
@@ -140,11 +140,12 @@ describe('Snapshotter', () => {
cpusMock.mockReturnValue(fakeCpu as any);
- const config = new Configuration({ maxUsedCpuRatio: 0.5 });
- const snapshotter = new Snapshotter({ config });
+ const noop = () => {};
+ serviceLocator.setConfiguration(new Configuration({ maxUsedCpuRatio: 0.5 }));
+ const snapshotter = new Snapshotter();
// do not initialize the event intervals as we will fire them manually
const spy = vitest.spyOn(LocalEventManager.prototype, 'init').mockImplementation(async () => {});
- const events = config.getEventManager() as LocalEventManager;
+ const events = serviceLocator.getEventManager() as LocalEventManager;
// Establish a baseline before the Snapshotter starts listening for events.
await events.emitSystemInfoEvent(noop);
@@ -217,11 +218,11 @@ describe('Snapshotter', () => {
childProcessesBytes: toBytes(1000),
} as MemoryInfo;
vitest.spyOn(utils, 'getMemoryInfo').mockResolvedValue(memoryData);
- const config = new Configuration({ availableMemoryRatio: 1 });
- const snapshotter = new Snapshotter({ config, maxUsedMemoryRatio: 0.5 });
+ serviceLocator.setConfiguration(new Configuration({ availableMemoryRatio: 1 }));
+ const snapshotter = new Snapshotter({ maxUsedMemoryRatio: 0.5 });
// do not initialize the event intervals as we will fire them manually
vitest.spyOn(LocalEventManager.prototype, 'init').mockImplementation(async () => {});
- const events = config.getEventManager() as LocalEventManager;
+ const events = serviceLocator.getEventManager() as LocalEventManager;
await snapshotter.start();
await events.emitSystemInfoEvent(noop);
@@ -248,10 +249,10 @@ describe('Snapshotter', () => {
test('correctly logs critical memory overload', async () => {
vitest.spyOn(utils, 'getMemoryInfo').mockResolvedValueOnce({ totalBytes: toBytes(10000) } as MemoryInfo);
- const config = new Configuration({ availableMemoryRatio: 1 });
- const snapshotter = new Snapshotter({ config, maxUsedMemoryRatio: 0.5 });
+ serviceLocator.setConfiguration(new Configuration({ availableMemoryRatio: 1 }));
+ const snapshotter = new Snapshotter({ maxUsedMemoryRatio: 0.5 });
- const eventManager = config.getEventManager() as LocalEventManager;
+ const eventManager = serviceLocator.getEventManager() as LocalEventManager;
await snapshotter.start();
const warningSpy = vitest.spyOn(snapshotter.log, 'warning').mockImplementation(() => {});
@@ -278,7 +279,7 @@ describe('Snapshotter', () => {
test('correctly marks clientOverloaded', () => {
// mock client data
- const apifyClient = Configuration.getStorageClient();
+ const apifyClient = serviceLocator.getStorageClient();
const oldStats = apifyClient.stats;
apifyClient.stats = {} as any;
apifyClient.stats!.rateLimitErrors = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
@@ -309,16 +310,15 @@ describe('Snapshotter', () => {
test('.get[.*]Sample limits amount of samples', async () => {
const SAMPLE_SIZE_MILLIS = 120;
- const config = new Configuration({ systemInfoIntervalMillis: 10 });
+ serviceLocator.setConfiguration(new Configuration({ systemInfoIntervalMillis: 10 }));
const snapshotter = new Snapshotter({
eventLoopSnapshotIntervalSecs: 0.01,
- config,
});
await snapshotter.start();
- await config.getEventManager().init();
+ await serviceLocator.getEventManager().init();
await sleep(1.5e3);
await snapshotter.stop();
- await config.getEventManager().close();
+ await serviceLocator.getEventManager().close();
const memorySnapshots = snapshotter.getMemorySample();
const eventLoopSnapshots = snapshotter.getEventLoopSample();
const memorySample = snapshotter.getMemorySample(SAMPLE_SIZE_MILLIS);
diff --git a/test/core/browser_launchers/playwright_launcher.test.ts b/test/core/browser_launchers/playwright_launcher.test.ts
index c9383809d..287677359 100644
--- a/test/core/browser_launchers/playwright_launcher.test.ts
+++ b/test/core/browser_launchers/playwright_launcher.test.ts
@@ -5,7 +5,13 @@ import type { AddressInfo } from 'node:net';
import path from 'node:path';
import util from 'node:util';
-import { BrowserLauncher, Configuration, launchPlaywright, PlaywrightLauncher } from '@crawlee/playwright';
+import {
+ BrowserLauncher,
+ Configuration,
+ launchPlaywright,
+ PlaywrightLauncher,
+ serviceLocator,
+} from '@crawlee/playwright';
// @ts-expect-error no types
import basicAuthParser from 'basic-auth-parser';
import type { Browser, BrowserType } from 'playwright';
@@ -14,7 +20,6 @@ import portastic from 'portastic';
import { createProxy } from 'proxy';
import { runExampleComServer } from 'test/shared/_helper.js';
-let prevEnvHeadless: boolean;
let proxyServer: Server;
let proxyPort: number;
const proxyAuth = { scheme: 'Basic', username: 'username', password: 'password' };
@@ -25,11 +30,11 @@ let server: Server;
let serverAddress = 'http://localhost:';
// Setup local proxy server for the tests
-beforeAll(async () => {
- const config = Configuration.getGlobalConfig();
- prevEnvHeadless = config.get('headless');
- config.set('headless', true);
+beforeEach(() => {
+ serviceLocator.setConfiguration(new Configuration({ headless: true }));
+});
+beforeAll(async () => {
[server, port] = await runExampleComServer();
serverAddress += port;
@@ -66,8 +71,6 @@ beforeAll(async () => {
});
afterAll(async () => {
- Configuration.getGlobalConfig().set('headless', prevEnvHeadless);
-
server.close();
if (proxyServer) await util.promisify(proxyServer.close).bind(proxyServer)();
}, 5000);
@@ -118,12 +121,10 @@ describe('launchPlaywright()', () => {
describe('headful mode', () => {
let browser: Browser;
- beforeAll(() => {
- // Test headless parameter
- Configuration.getGlobalConfig().set('headless', false);
- });
-
beforeEach(async () => {
+ // Test headless parameter - reset first since outer beforeEach already set configuration
+ serviceLocator.reset();
+ serviceLocator.setConfiguration(new Configuration({ headless: false }));
browser = await launchPlaywright({
launchOptions: { headless: true, timeout: 60e3 },
proxyUrl: `http://username:[email protected]:${proxyPort}`,
@@ -134,10 +135,6 @@ describe('launchPlaywright()', () => {
if (browser) await browser.close();
});
- afterAll(() => {
- Configuration.getGlobalConfig().set('headless', true);
- });
-
test('opens a webpage via proxy with authentication', async () => {
const page = await browser.newPage();
diff --git a/test/core/crawlers/adaptive_playwright_crawler.test.ts b/test/core/crawlers/adaptive_playwright_crawler.test.ts
index 36497ec8b..1bae3cba3 100644
--- a/test/core/crawlers/adaptive_playwright_crawler.test.ts
+++ b/test/core/crawlers/adaptive_playwright_crawler.test.ts
@@ -1,7 +1,7 @@
import type { Server } from 'node:http';
import type { AddressInfo } from 'node:net';
-import { Configuration, type Dictionary, EventType, KeyValueStore } from '@crawlee/core';
+import { Configuration, type Dictionary, EventType, KeyValueStore, serviceLocator } from '@crawlee/core';
import type {
AdaptivePlaywrightCrawlerContext,
AdaptivePlaywrightCrawlerOptions,
@@ -583,7 +583,7 @@ describe('AdaptivePlaywrightCrawler', () => {
await crawler.run();
// Now emit a PERSIST_STATE event to trigger state persistence
- const events = Configuration.getEventManager();
+ const events = serviceLocator.getEventManager();
events.emit(EventType.PERSIST_STATE);
// Wait a bit for the event to be processed
diff --git a/test/core/crawlers/basic_crawler.test.ts b/test/core/crawlers/basic_crawler.test.ts
index bb90b707b..3dbe32912 100644
--- a/test/core/crawlers/basic_crawler.test.ts
+++ b/test/core/crawlers/basic_crawler.test.ts
@@ -2,7 +2,7 @@ import { readFile, rm } from 'node:fs/promises';
import type { Server } from 'node:http';
import http from 'node:http';
import type { AddressInfo } from 'node:net';
-
+import log from '@apify/log';
import type { EnqueueLinksOptions, ErrorHandler, RequestHandler, RequestOptions, Source } from '@crawlee/basic';
import {
BasicCrawler,
@@ -18,6 +18,7 @@ import {
RequestQueue,
RequestValidationError,
Router,
+ serviceLocator,
} from '@crawlee/basic';
import { RequestState, SessionPool, Statistics } from '@crawlee/core';
import type { Dictionary } from '@crawlee/utils';
@@ -29,16 +30,11 @@ import type { Mock } from 'vitest';
import { afterAll, beforeAll, beforeEach, describe, expect, test, vitest } from 'vitest';
import { z } from 'zod';
-import log from '@apify/log';
-
import { startExpressAppPromise } from 'test/shared/_helper.js';
-import { MemoryStorageEmulator } from 'test/shared/MemoryStorageEmulator.js';
describe('BasicCrawler', () => {
let logLevel: number;
const localStorageEmulator = new MemoryStorageEmulator();
- const events = Configuration.getEventManager();
-
const HOSTNAME = '127.0.0.1';
let port: number;
let server: Server;
@@ -527,7 +523,7 @@ describe('BasicCrawler', () => {
const processed: { url: string }[] = [];
const requestList = await RequestList.open('reqList', sources);
const requestHandler: RequestHandler = async ({ request }) => {
- if (request.url.endsWith('200')) events.emit(event);
+ if (request.url.endsWith('200')) serviceLocator.getEventManager().emit(event);
processed.push({ url: request.url });
};
@@ -932,7 +928,10 @@ describe('BasicCrawler', () => {
];
const processed: Dictionary<Request> = {};
const requestList = await RequestList.open(null, sources);
- const requestQueue = new RequestQueue({ id: 'xxx', client: Configuration.getStorageClient() });
+ const requestQueue = new RequestQueue(
+ { id: 'xxx', client: serviceLocator.getStorageClient() },
+ serviceLocator.getConfiguration(),
+ );
const requestHandler: RequestHandler = async ({ request }) => {
await sleep(10);
@@ -1003,7 +1002,10 @@ describe('BasicCrawler', () => {
});
test('should say that task is not ready requestList is not set and requestQueue is empty', async () => {
- const requestQueue = new RequestQueue({ id: 'xxx', client: Configuration.getStorageClient() });
+ const requestQueue = new RequestQueue(
+ { id: 'xxx', client: serviceLocator.getStorageClient() },
+ serviceLocator.getConfiguration(),
+ );
requestQueue.isEmpty = async () => Promise.resolve(true);
const crawler = new BasicCrawler({
@@ -1016,7 +1018,10 @@ describe('BasicCrawler', () => {
});
test('should be possible to override isFinishedFunction and isTaskReadyFunction of underlying AutoscaledPool', async () => {
- const requestQueue = new RequestQueue({ id: 'xxx', client: Configuration.getStorageClient() });
+ const requestQueue = new RequestQueue(
+ { id: 'xxx', client: serviceLocator.getStorageClient() },
+ serviceLocator.getConfiguration(),
+ );
const processed: Request[] = [];
const queue: Request[] = [];
let isFinished = false;
@@ -1090,7 +1095,10 @@ describe('BasicCrawler', () => {
});
test('keepAlive', async () => {
- const requestQueue = new RequestQueue({ id: 'xxx', client: Configuration.getStorageClient() });
+ const requestQueue = new RequestQueue(
+ { id: 'xxx', client: serviceLocator.getStorageClient() },
+ serviceLocator.getConfiguration(),
+ );
const processed: Request[] = [];
const queue: Request[] = [];
@@ -1190,8 +1198,11 @@ describe('BasicCrawler', () => {
expect(await requestList.isEmpty()).toBe(false);
});
- test('should derive handledRequestCount from Statistics', async () => {
- const requestQueue = new RequestQueue({ id: 'id', client: Configuration.getStorageClient() });
+ test('should load handledRequestCount from storages', async () => {
+ const requestQueue = new RequestQueue(
+ { id: 'id', client: serviceLocator.getStorageClient() },
+ serviceLocator.getConfiguration(),
+ );
requestQueue.isEmpty = async () => false;
requestQueue.isFinished = async () => false;
@@ -1523,7 +1534,7 @@ describe('BasicCrawler', () => {
it('should destroy Session pool after it is finished', async () => {
const url = 'https://example.com';
const requestList = await RequestList.open({ sources: [{ url }] });
- events.off(EventType.PERSIST_STATE);
+ serviceLocator.getEventManager().off(EventType.PERSIST_STATE);
const crawler = new BasicCrawler({
requestList,
@@ -1537,8 +1548,14 @@ describe('BasicCrawler', () => {
failedRequestHandler: async () => {},
});
+ // @ts-expect-error Accessing private prop
+ crawler._loadHandledRequestCount = () => {
+ expect(crawler.sessionPool).toBeDefined();
+ expect(serviceLocator.getEventManager().listenerCount(EventType.PERSIST_STATE)).toEqual(1);
+ };
+
await crawler.run();
- expect(events.listenerCount(EventType.PERSIST_STATE)).toEqual(0);
+ expect(serviceLocator.getEventManager().listenerCount(EventType.PERSIST_STATE)).toEqual(0);
// @ts-expect-error private symbol
expect(crawler.sessionPool.maxPoolSize).toEqual(10);
});
@@ -2484,8 +2501,8 @@ describe('BasicCrawler', () => {
});
test("Crawlers with different Configurations don't share Datasets", async () => {
- const crawlerA = new BasicCrawler({}, new Configuration({ persistStorage: false }));
- const crawlerB = new BasicCrawler({}, new Configuration({ persistStorage: false }));
+ const crawlerA = new BasicCrawler({ configuration: new Configuration({ persistStorage: false }) });
+ const crawlerB = new BasicCrawler({ configuration: new Configuration({ persistStorage: false }) });
await crawlerA.pushData(getPayload('A'));
await crawlerB.pushData(getPayload('B'));
@@ -2496,14 +2513,14 @@ describe('BasicCrawler', () => {
});
test('Crawlers with different Configurations run separately', async () => {
- const crawlerA = new BasicCrawler(
- { requestHandler: () => {} },
- new Configuration({ persistStorage: false }),
- );
- const crawlerB = new BasicCrawler(
- { requestHandler: () => {} },
- new Configuration({ persistStorage: false }),
- );
+ const crawlerA = new BasicCrawler({
+ requestHandler: () => {},
+ configuration: new Configuration({ persistStorage: false }),
+ });
+ const crawlerB = new BasicCrawler({
+ requestHandler: () => {},
+ configuration: new Configuration({ persistStorage: false }),
+ });
await crawlerA.run([{ url: `http://${HOSTNAME}:${port}` }]);
await crawlerB.run([{ url: `http://${HOSTNAME}:${port}` }]);
@@ -2516,15 +2533,17 @@ describe('BasicCrawler', () => {
const getGlobalConfigSpy = vitest.spyOn(Configuration, 'getGlobalConfig');
const configA = new Configuration({ persistStorage: false });
- const crawlerA = new BasicCrawler({ requestHandler: () => {} }, configA);
+ const crawlerA = new BasicCrawler({ requestHandler: () => {}, configuration: configA });
const configB = new Configuration({ persistStorage: false });
- const crawlerB = new BasicCrawler({ requestHandler: () => {} }, configB);
+ const crawlerB = new BasicCrawler({ requestHandler: () => {}, configuration: configB });
await crawlerA.run([{ url: `http://${HOSTNAME}:${port}` }]);
await crawlerB.run([{ url: `http://${HOSTNAME}:${port}` }]);
expect(getGlobalConfigSpy.mock.calls.length).toBe(0);
+ // @ts-expect-error Accessing protected property for testing
expect(crawlerA.requestQueue?.config).toBe(configA);
+ // @ts-expect-error Accessing protected property for testing
expect(crawlerB.requestQueue?.config).toBe(configB);
});
});
diff --git a/test/core/crawlers/statistics.test.ts b/test/core/crawlers/statistics.test.ts
index 9de4ed857..f9b6dfc13 100644
--- a/test/core/crawlers/statistics.test.ts
+++ b/test/core/crawlers/statistics.test.ts
@@ -1,4 +1,4 @@
-import { Configuration, EventType, Statistics } from '@crawlee/core';
+import { Configuration, EventType, serviceLocator, Statistics } from '@crawlee/core';
import type { Dictionary } from '@crawlee/utils';
import { MemoryStorageEmulator } from 'test/shared/MemoryStorageEmulator.js';
@@ -9,8 +9,6 @@ describe('Statistics', () => {
let stats: Statistics;
const localStorageEmulator = new MemoryStorageEmulator();
- const events = Configuration.getEventManager();
-
beforeAll(async () => {
vitest.useFakeTimers();
});
@@ -21,7 +19,7 @@ describe('Statistics', () => {
});
afterEach(async () => {
- events.off(EventType.PERSIST_STATE);
+ serviceLocator.getEventManager().off(EventType.PERSIST_STATE);
stats = null as any;
});
@@ -157,6 +155,7 @@ describe('Statistics', () => {
});
test('should remove persist state event listener', async () => {
+ const events = serviceLocator.getEventManager();
await stats.startCapturing();
expect(events.listenerCount(EventType.PERSIST_STATE)).toEqual(1);
await stats.stopCapturing();
@@ -180,7 +179,7 @@ describe('Statistics', () => {
// @ts-expect-error Accessing private prop
const setValueSpy = vitest.spyOn(stats.keyValueStore, 'setValue');
- events.emit(EventType.PERSIST_STATE);
+ serviceLocator.getEventManager().emit(EventType.PERSIST_STATE);
// TODO: these properties don't exist on the calculate return type
// @ts-expect-error Incorrect types?
diff --git a/test/core/enqueue_links/click_elements.test.ts b/test/core/enqueue_links/click_elements.test.ts
index 8331a7ad7..e719078b1 100644
--- a/test/core/enqueue_links/click_elements.test.ts
+++ b/test/core/enqueue_links/click_elements.test.ts
@@ -10,6 +10,7 @@ import {
puppeteerClickElements,
puppeteerUtils,
RequestQueue,
+ serviceLocator,
} from 'crawlee';
import type { Browser as PWBrowser, Page as PWPage } from 'playwright';
// @ts-ignore This only throws when compiled against puppeteer 25+ (ESM only), we only import types, so its alllll gooooood
@@ -24,11 +25,11 @@ function isPlaywrightBrowser(browser: PPBrowser | PWBrowser): browser is PWBrows
return (browser as PWBrowser).browserType !== undefined;
}
-const apifyClient = Configuration.getStorageClient();
+const apifyClient = serviceLocator.getStorageClient();
function createRequestQueueMock() {
const enqueued: Source[] = [];
- const requestQueue = new RequestQueue({ id: 'xxx', client: apifyClient });
+ const requestQueue = new RequestQueue({ id: 'xxx', client: apifyClient }, serviceLocator.getConfiguration());
// @ts-expect-error Override method for testing
requestQueue.addRequests = async function (requests) {
@@ -116,7 +117,10 @@ testCases.forEach(({ caseName, launchBrowser, clickElements, utils }) => {
test('accepts forefront option', async () => {
const addedRequests: { request: Source; options?: RequestQueueOperationOptions }[] = [];
- const requestQueue = new RequestQueue({ id: 'xxx', client: Configuration.getStorageClient() });
+ const requestQueue = new RequestQueue(
+ { id: 'xxx', client: serviceLocator.getStorageClient() },
+ serviceLocator.getConfiguration(),
+ );
requestQueue.addRequests = async (requests, options) => {
for await (const request of requests) {
addedRequests.push({ request: typeof request === 'string' ? { url: request } : request, options });
diff --git a/test/core/enqueue_links/enqueue_links.test.ts b/test/core/enqueue_links/enqueue_links.test.ts
index 9dce392ff..e67992017 100644
--- a/test/core/enqueue_links/enqueue_links.test.ts
+++ b/test/core/enqueue_links/enqueue_links.test.ts
@@ -9,6 +9,7 @@ import {
launchPuppeteer,
Request,
RequestQueue,
+ serviceLocator,
} from '@crawlee/puppeteer';
import { type CheerioRoot, RobotsTxtFile } from '@crawlee/utils';
import { load } from 'cheerio';
@@ -18,7 +19,7 @@ import type { Browser as PuppeteerBrowser, Page as PuppeteerPage } from 'puppete
import log from '@apify/log';
-const apifyClient = Configuration.getStorageClient();
+const apifyClient = serviceLocator.getStorageClient();
const HTML = `
<html>
@@ -49,7 +50,7 @@ const HTML = `
function createRequestQueueMock() {
const enqueued: Source[] = [];
- const requestQueue = new RequestQueue({ id: 'xxx', client: apifyClient });
+ const requestQueue = new RequestQueue({ id: 'xxx', client: apifyClient }, serviceLocator.getConfiguration());
// @ts-expect-error Override method for testing
requestQueue.addRequests = async function (requests) {
@@ -1025,7 +1026,10 @@ describe('enqueueLinks()', () => {
test('accepts forefront option', async () => {
const enqueued: { request: Source; options?: RequestQueueOperationOptions }[] = [];
- const requestQueue = new RequestQueue({ id: 'xxx', client: apifyClient });
+ const requestQueue = new RequestQueue(
+ { id: 'xxx', client: apifyClient },
+ serviceLocator.getConfiguration(),
+ );
requestQueue.addRequests = async (requests, options) => {
// copy the requests to the enqueued list, along with options that were passed to addRequests,
@@ -1054,7 +1058,10 @@ describe('enqueueLinks()', () => {
test('accepts waitForAllRequestsToBeAdded option', async () => {
const enqueued: { request: string | Source; options?: AddRequestsBatchedOptions }[] = [];
- const requestQueue = new RequestQueue({ id: 'xxx', client: apifyClient });
+ const requestQueue = new RequestQueue(
+ { id: 'xxx', client: apifyClient },
+ serviceLocator.getConfiguration(),
+ );
requestQueue.addRequestsBatched = async (requests, options) => {
// copy the requests to the enqueued list, along with options that were passed to addRequests,
diff --git a/test/core/request_list.test.ts b/test/core/request_list.test.ts
index 045fc6363..b34d95e97 100644
--- a/test/core/request_list.test.ts
+++ b/test/core/request_list.test.ts
@@ -6,6 +6,7 @@ import {
ProxyConfiguration,
Request,
RequestList,
+ serviceLocator,
} from '@crawlee/core';
import { sleep } from '@crawlee/utils';
import { MemoryStorageEmulator } from 'test/shared/MemoryStorageEmulator.js';
@@ -48,8 +49,6 @@ beforeEach(async () => {
describe('RequestList', () => {
let ll: number;
const emulator = new MemoryStorageEmulator();
- const events = Configuration.getEventManager();
-
beforeAll(() => {
ll = log.getLevel();
log.setLevel(log.LEVELS.ERROR);
@@ -489,7 +488,7 @@ describe('RequestList', () => {
// Persist state.
setValueSpy.mockResolvedValueOnce();
- events.emit(EventType.PERSIST_STATE);
+ serviceLocator.getEventManager().emit(EventType.PERSIST_STATE);
await sleep(20);
expect(requestList.isStatePersisted).toBe(true);
@@ -499,7 +498,7 @@ describe('RequestList', () => {
await requestList.markRequestHandled(request2!);
expect(requestList.isStatePersisted).toBe(false);
setValueSpy.mockResolvedValueOnce();
- events.emit(EventType.PERSIST_STATE);
+ serviceLocator.getEventManager().emit(EventType.PERSIST_STATE);
await sleep(20);
expect(requestList.isStatePersisted).toBe(true);
diff --git a/test/core/session_pool/session_pool.test.ts b/test/core/session_pool/session_pool.test.ts
index 01ee31b49..c3b134d5b 100644
--- a/test/core/session_pool/session_pool.test.ts
+++ b/test/core/session_pool/session_pool.test.ts
@@ -1,4 +1,4 @@
-import { Configuration, EventType, KeyValueStore, Session, SessionPool } from '@crawlee/core';
+import { EventType, KeyValueStore, serviceLocator, Session, SessionPool } from '@crawlee/core';
import { entries } from '@crawlee/utils';
import { MemoryStorageEmulator } from 'test/shared/MemoryStorageEmulator.js';
@@ -7,7 +7,6 @@ import { Log } from '@apify/log';
describe('SessionPool - testing session pool', () => {
let sessionPool: SessionPool;
const localStorageEmulator = new MemoryStorageEmulator();
- const events = Configuration.getEventManager();
beforeEach(async () => {
await localStorageEmulator.init();
@@ -15,7 +14,7 @@ describe('SessionPool - testing session pool', () => {
});
afterEach(async () => {
- events.off(EventType.PERSIST_STATE);
+ serviceLocator.getEventManager().off(EventType.PERSIST_STATE);
});
afterAll(async () => {
@@ -245,7 +244,7 @@ describe('SessionPool - testing session pool', () => {
// @ts-expect-error private symbol
expect(sessionPool.sessions.length).toBe(1);
- events.emit(EventType.PERSIST_STATE);
+ serviceLocator.getEventManager().emit(EventType.PERSIST_STATE);
await new Promise<void>((resolve) => {
const interval = setInterval(async () => {
@@ -359,6 +358,7 @@ describe('SessionPool - testing session pool', () => {
});
it('should remove persist state event listener', async () => {
+ const events = serviceLocator.getEventManager();
expect(events.listenerCount(EventType.PERSIST_STATE)).toEqual(1);
await sessionPool.teardown();
expect(events.listenerCount(EventType.PERSIST_STATE)).toEqual(0);
diff --git a/test/core/storages/dataset.test.ts b/test/core/storages/dataset.test.ts
index d66f1ea2a..76842ba23 100644
--- a/test/core/storages/dataset.test.ts
+++ b/test/core/storages/dataset.test.ts
@@ -1,4 +1,4 @@
-import { checkAndSerialize, chunkBySize, Configuration, Dataset, KeyValueStore } from '@crawlee/core';
+import { checkAndSerialize, chunkBySize, Configuration, Dataset, KeyValueStore, serviceLocator } from '@crawlee/core';
import type { Dictionary } from '@crawlee/utils';
import { MemoryStorageEmulator } from 'test/shared/MemoryStorageEmulator.js';
@@ -15,7 +15,7 @@ afterAll(async () => {
});
describe('dataset', () => {
- const storageClient = Configuration.getStorageClient();
+ const storageClient = serviceLocator.getStorageClient();
beforeEach(async () => {
vitest.clearAllMocks();
diff --git a/test/core/storages/key_value_store.test.ts b/test/core/storages/key_value_store.test.ts
index c9a107555..b09d692e3 100644
--- a/test/core/storages/key_value_store.test.ts
+++ b/test/core/storages/key_value_store.test.ts
@@ -1,6 +1,6 @@
import { PassThrough } from 'node:stream';
-import { Configuration, KeyValueStore, maybeStringify } from '@crawlee/core';
+import { Configuration, KeyValueStore, maybeStringify, serviceLocator } from '@crawlee/core';
import type { Dictionary } from '@crawlee/utils';
import { MemoryStorageEmulator } from 'test/shared/MemoryStorageEmulator.js';
@@ -15,7 +15,7 @@ afterAll(async () => {
});
describe('KeyValueStore', () => {
- const client = Configuration.getStorageClient();
+ const client = serviceLocator.getStorageClient();
beforeEach(async () => {
vitest.clearAllMocks();
diff --git a/test/core/storages/request_queue.test.ts b/test/core/storages/request_queue.test.ts
index a15d44828..5e7c27cda 100644
--- a/test/core/storages/request_queue.test.ts
+++ b/test/core/storages/request_queue.test.ts
@@ -8,6 +8,7 @@ import {
Request,
RequestQueueV1 as RequestQueue,
RequestQueueV2,
+ serviceLocator,
STORAGE_CONSISTENCY_DELAY_MILLIS,
} from '@crawlee/core';
import { sleep } from '@crawlee/utils';
@@ -37,14 +38,14 @@ beforeEach(async () => {
});
describe('RequestQueue remote', () => {
- const storageClient = Configuration.getStorageClient();
+ const storageClient = serviceLocator.getStorageClient();
beforeEach(() => {
vitest.clearAllMocks();
});
test('should work', async () => {
- const queue = new RequestQueue({ id: 'some-id', client: storageClient });
+ const queue = new RequestQueue({ id: 'some-id', client: storageClient }, serviceLocator.getConfiguration());
const firstResolveValue = {
requestId: 'a',
wasAlreadyHandled: false,
@@ -201,7 +202,10 @@ describe('RequestQueue remote', () => {
});
test('addRequests', async () => {
- const queue = new RequestQueue({ id: 'batch-requests', client: storageClient });
+ const queue = new RequestQueue(
+ { id: 'batch-requests', client: storageClient },
+ serviceLocator.getConfiguration(),
+ );
const mockAddRequests = vitest.spyOn(queue.client, 'batchAddRequests');
const requestOptions = { url: 'http://example.com/a' };
@@ -350,7 +354,7 @@ describe('RequestQueue remote', () => {
});
test('should cache new requests locally', async () => {
- const queue = new RequestQueue({ id: 'some-id', client: storageClient });
+ const queue = new RequestQueue({ id: 'some-id', client: storageClient }, serviceLocator.getConfiguration());
const requestA = new Request({ url: 'http://example.com/a' });
const requestB = new Request({ url: 'http://example.com/a' }); // Has same uniqueKey as A
@@ -381,7 +385,7 @@ describe('RequestQueue remote', () => {
});
test('should cache requests locally with info if request was already handled', async () => {
- const queue = new RequestQueue({ id: 'some-id', client: storageClient });
+ const queue = new RequestQueue({ id: 'some-id', client: storageClient }, serviceLocator.getConfiguration());
const requestX = new Request({ url: 'http://example.com/x' });
const requestY = new Request({ url: 'http://example.com/x' }); // Has same uniqueKey as X
@@ -412,7 +416,7 @@ describe('RequestQueue remote', () => {
});
test('should cache requests from queue head', async () => {
- const queue = new RequestQueue({ id: 'some-id', client: storageClient });
+ const queue = new RequestQueue({ id: 'some-id', client: storageClient }, serviceLocator.getConfiguration());
// Query queue head with request A
const listHeadMock = vitest.spyOn(queue.client, 'listHead');
@@ -440,7 +444,10 @@ describe('RequestQueue remote', () => {
});
test('should handle situation when newly created request is not available yet', async () => {
- const queue = new RequestQueue({ id: 'some-id', name: 'some-queue', client: storageClient });
+ const queue = new RequestQueue(
+ { id: 'some-id', name: 'some-queue', client: storageClient },
+ serviceLocator.getConfiguration(),
+ );
const listHeadMock = vitest.spyOn(queue.client, 'listHead');
const requestA = new Request({ url: 'http://example.com/a' });
@@ -491,7 +498,7 @@ describe('RequestQueue remote', () => {
});
test('should not add handled request to queue head dict', async () => {
- const queue = new RequestQueue({ id: 'some-id', client: storageClient });
+ const queue = new RequestQueue({ id: 'some-id', client: storageClient }, serviceLocator.getConfiguration());
const requestA = new Request({ url: 'http://example.com/a' });
@@ -521,7 +528,7 @@ describe('RequestQueue remote', () => {
});
test('should accept plain object in addRequest()', async () => {
- const queue = new RequestQueue({ id: 'some-id', client: storageClient });
+ const queue = new RequestQueue({ id: 'some-id', client: storageClient }, serviceLocator.getConfiguration());
const addRequestMock = vitest.spyOn(queue.client, 'addRequest');
addRequestMock.mockResolvedValueOnce({
requestId: 'xxx',
@@ -536,7 +543,7 @@ describe('RequestQueue remote', () => {
});
test('should return correct handledCount', async () => {
- const queue = new RequestQueue({ id: 'id', client: storageClient });
+ const queue = new RequestQueue({ id: 'id', client: storageClient }, serviceLocator.getConfiguration());
const getMock = vitest.spyOn(queue.client, 'get');
getMock.mockResolvedValueOnce({
handledRequestCount: 33,
@@ -548,7 +555,10 @@ describe('RequestQueue remote', () => {
});
test('should always wait for a queue head to become consistent before marking queue as finished (hadMultipleClients = true)', async () => {
- const queue = new RequestQueue({ id: 'some-id', name: 'some-name', client: storageClient });
+ const queue = new RequestQueue(
+ { id: 'some-id', name: 'some-name', client: storageClient },
+ serviceLocator.getConfiguration(),
+ );
// Return head with modifiedAt = now so it will retry the call.
const listHeadMock = vitest.spyOn(queue.client, 'listHead');
@@ -574,7 +584,10 @@ describe('RequestQueue remote', () => {
test('should always wait for a queue head to become consistent before marking queue as finished (hadMultipleClients = false)', async () => {
const queueId = 'some-id';
- const queue = new RequestQueue({ id: queueId, name: 'some-name', client: storageClient });
+ const queue = new RequestQueue(
+ { id: queueId, name: 'some-name', client: storageClient },
+ serviceLocator.getConfiguration(),
+ );
expect(queue.assumedTotalCount).toBe(0);
expect(queue.assumedHandledCount).toBe(0);
@@ -740,7 +753,10 @@ describe('RequestQueue remote', () => {
});
test('getInfo() should work', async () => {
- const queue = new RequestQueue({ id: 'some-id', name: 'some-name', client: storageClient });
+ const queue = new RequestQueue(
+ { id: 'some-id', name: 'some-name', client: storageClient },
+ serviceLocator.getConfiguration(),
+ );
const expected = {
id: 'WkzbQMuFYuamGv3YF',
@@ -765,7 +781,10 @@ describe('RequestQueue remote', () => {
});
test('drop() works', async () => {
- const queue = new RequestQueue({ id: 'some-id', name: 'some-name', client: storageClient });
+ const queue = new RequestQueue(
+ { id: 'some-id', name: 'some-name', client: storageClient },
+ serviceLocator.getConfiguration(),
+ );
const deleteMock = vitest.spyOn(queue.client, 'delete').mockResolvedValueOnce(undefined);
await queue.drop();
diff --git a/test/core/storages/utils.test.ts b/test/core/storages/utils.test.ts
index 93d009830..ff9bf0d24 100644
--- a/test/core/storages/utils.test.ts
+++ b/test/core/storages/utils.test.ts
@@ -1,18 +1,16 @@
import type { Dictionary } from '@crawlee/core';
-import { Configuration, KeyValueStore, purgeDefaultStorages, useState } from '@crawlee/core';
+import { Configuration, KeyValueStore, purgeDefaultStorages, serviceLocator, useState } from '@crawlee/core';
import type { StorageClient } from '@crawlee/types';
+
import { MemoryStorageEmulator } from 'test/shared/MemoryStorageEmulator.js';
describe('useState', () => {
const emulator = new MemoryStorageEmulator();
- beforeAll(async () => {
- Configuration.getGlobalConfig().set('persistStateIntervalMillis', 1e3);
- });
-
beforeEach(async () => {
await emulator.init();
+ serviceLocator.setConfiguration(new Configuration({ persistStateIntervalMillis: 1e3 }));
});
afterAll(async () => {
@@ -47,7 +45,7 @@ describe('useState', () => {
state.hello = 'foo';
state.foo = ['fizz'];
- const manager = Configuration.getEventManager();
+ const manager = serviceLocator.getEventManager();
await manager.init();
diff --git a/test/shared/MemoryStorageEmulator.ts b/test/shared/MemoryStorageEmulator.ts
index a0cad7003..c81876efe 100644
--- a/test/shared/MemoryStorageEmulator.ts
+++ b/test/shared/MemoryStorageEmulator.ts
@@ -1,7 +1,7 @@
import { resolve } from 'node:path';
import { MemoryStorage } from '@crawlee/memory-storage';
-import { Configuration } from 'crawlee';
+import { Configuration, serviceLocator } from 'crawlee';
import { ensureDir } from 'fs-extra';
import log from '@apify/log';
@@ -22,7 +22,7 @@ export class MemoryStorageEmulator extends StorageEmulator {
this.storage = new MemoryStorage({ localDataDirectory: localStorageDir, persistStorage, writeMetadata: false });
- Configuration.getGlobalConfig().useStorageClient(this.storage);
+ serviceLocator.setStorageClient(this.storage);
log.debug(`Initialized emulated memory storage in folder ${localStorageDir}`);
}
diff --git a/test/shared/StorageEmulator.ts b/test/shared/StorageEmulator.ts
index 55746dacc..da62bcd54 100644
--- a/test/shared/StorageEmulator.ts
+++ b/test/shared/StorageEmulator.ts
@@ -1,12 +1,12 @@
import { rm } from 'node:fs/promises';
-import { StorageManager } from '@crawlee/core';
+import { serviceLocator } from '@crawlee/core';
export abstract class StorageEmulator {
protected localStorageDirectories: string[] = [];
async init(options?: Record<PropertyKey, any>): Promise<void> {
- StorageManager.clearCache();
+ serviceLocator.reset();
}
async destroy() {
@@ -15,6 +15,6 @@ export abstract class StorageEmulator {
});
await Promise.all(promises);
- StorageManager.clearCache();
+ serviceLocator.clearStorageManagerCache();
}
}
diff --git a/test/vitest.setup.ts b/test/vitest.setup.ts
new file mode 100644
index 000000000..502c8d1ba
--- /dev/null
+++ b/test/vitest.setup.ts
@@ -0,0 +1,7 @@
+import { beforeEach } from 'vitest';
+
+import { serviceLocator } from '../packages/core/src/service_locator.js';
+
+beforeEach(() => {
+ serviceLocator.reset();
+});
diff --git a/vitest.config.mts b/vitest.config.mts
index d2fe69a59..b31ed8f6f 100644
--- a/vitest.config.mts
+++ b/vitest.config.mts
@@ -18,6 +18,7 @@ const baseConfig = defineConfig({
},
test: {
globals: true,
+ setupFiles: ['./test/vitest.setup.ts'],
coverage: {
provider: 'v8',
reporter: ['text', 'lcov', 'cobertura'],