Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

chore: Rename client load signal to storageBackend (#4023)

Jan Buchar8e3194b91ac6
diff --git a/docs/guides/scaling_crawlers.mdx b/docs/guides/scaling_crawlers.mdx
index 98ab04ad7..5a73722ce 100644
--- a/docs/guides/scaling_crawlers.mdx
+++ b/docs/guides/scaling_crawlers.mdx
@@ -122,7 +122,7 @@ This option can be set by specifying [`maxRequestsPerMinute`](#maxrequestspermin
## Load signals
-Whether the machine counts as overloaded is decided by *load signals*. Four are built in — memory, event loop, CPU and the storage client's rate-limit errors — and each is configured by its own bag under <ApiLink to="core/interface/ConcurrencySystemOptions#loadSignals">`loadSignals`</ApiLink>, carrying both its limits and the `overloadedRatio` at which it fires. If any signal reports overload, the system is overloaded and concurrency is held down.
+Whether the machine counts as overloaded is decided by *load signals*. Four are built in — memory, event loop, CPU and the storage backend's rate-limit errors — and each is configured by its own bag under <ApiLink to="core/interface/ConcurrencySystemOptions#loadSignals">`loadSignals`</ApiLink>, carrying both its limits and the `overloadedRatio` at which it fires. If any signal reports overload, the system is overloaded and concurrency is held down.
A signal we don't want watched at all can be switched **off** with `false`, which stops it being collected as well as evaluated (its entry in the reported status then simply reads as not overloaded):
@@ -136,7 +136,7 @@ We can also watch resources of our own by implementing <ApiLink to="core/interfa
{CustomLoadSignalSource}
</CodeBlock>
-Each built-in is a public class too — `MemoryLoadSignal`, `EventLoopLoadSignal`, `CpuLoadSignal`, `ClientLoadSignal` — taking exactly the bag its `loadSignals` key accepts. Constructing one directly is how we *wrap* a built-in rather than reimplement it (to hold its overload verdict for a while after the resource recovers, say): switch the original off with `cpu: false`, and pass a signal that delegates to the instance we built in `custom`.
+Each built-in is a public class too — `MemoryLoadSignal`, `EventLoopLoadSignal`, `CpuLoadSignal`, `StorageBackendLoadSignal` — taking exactly the bag its `loadSignals` key accepts. Constructing one directly is how we *wrap* a built-in rather than reimplement it (to hold its overload verdict for a while after the resource recovers, say): switch the original off with `cpu: false`, and pass a signal that delegates to the instance we built in `custom`.
<CodeBlock language="js">
{WrapLoadSignalSource}
@@ -144,7 +144,7 @@ Each built-in is a public class too — `MemoryLoadSignal`, `EventLoopLoadSignal
:::info Signal names are the keys of the reported status
-A signal's `name` is the key its verdict appears under in the reported status, so two signals cannot share one — a duplicate throws at construction time. That is why taking over a built-in name (`memInfo`, `eventLoopInfo`, `cpuInfo`, `clientInfo`) requires switching that built-in off.
+A signal's `name` is the key its verdict appears under in the reported status, so two signals cannot share one — a duplicate throws at construction time. That is why taking over a built-in name (`memInfo`, `eventLoopInfo`, `cpuInfo`, `storageBackendInfo`) requires switching that built-in off.
:::
diff --git a/docs/guides/scaling_crawlers_loadSignalsOff.ts b/docs/guides/scaling_crawlers_loadSignalsOff.ts
index 06f43eea7..aa982a094 100644
--- a/docs/guides/scaling_crawlers_loadSignalsOff.ts
+++ b/docs/guides/scaling_crawlers_loadSignalsOff.ts
@@ -3,7 +3,7 @@ import { ConcurrencySystem } from 'crawlee';
const concurrencySystem = new ConcurrencySystem({
loadSignals: {
// Our storage backend reports no rate-limit statistics, so stop polling it every second.
- client: false,
+ storageBackend: false,
eventLoop: { maxBlockedMillis: 100 },
},
});
diff --git a/docs/public-api/crawlee-core.api.md b/docs/public-api/crawlee-core.api.md
index 4b54d2473..c290738cf 100644
--- a/docs/public-api/crawlee-core.api.md
+++ b/docs/public-api/crawlee-core.api.md
@@ -146,38 +146,6 @@ export interface CalculatedStatistics {
requestTotalDurationMillis: number;
}
-// @public (undocumented)
-export interface ClientInfo {
- // (undocumented)
- actualRatio: number;
- // (undocumented)
- isOverloaded: boolean;
- // (undocumented)
- limitRatio: number;
-}
-
-// @public
-export class ClientLoadSignal implements LoadSignal {
- constructor(options?: ClientLoadSignalOptions);
- // (undocumented)
- getSample(sampleDurationMillis?: number): LoadSnapshot[];
- // (undocumented)
- readonly name = "clientInfo";
- // (undocumented)
- readonly overloadedRatio: number;
- // (undocumented)
- start(context: LoadSignalStartContext): Promise<void>;
- // (undocumented)
- stop(): Promise<void>;
-}
-
-// @public
-export interface ClientLoadSignalOptions {
- maxErrors?: number;
- overloadedRatio?: number;
- snapshotIntervalSecs?: number;
-}
-
// @public
export const coerceBoolean: z.ZodPreprocess<z.ZodBoolean>;
@@ -1027,13 +995,23 @@ export interface LoadSignal {
stop(): Promise<void>;
}
+// @public (undocumented)
+export interface LoadSignalInfo {
+ // (undocumented)
+ actualRatio: number;
+ // (undocumented)
+ isOverloaded: boolean;
+ // (undocumented)
+ limitRatio: number;
+}
+
// @public
export interface LoadSignalsOptions {
- client?: ClientLoadSignalOptions | false;
cpu?: CpuLoadSignalOptions | false;
custom?: LoadSignal[];
eventLoop?: EventLoopLoadSignalOptions | false;
memory?: MemoryLoadSignalOptions | false;
+ storageBackend?: StorageBackendLoadSignalOptions | false;
}
// @public
@@ -2034,6 +2012,28 @@ export interface StatisticState {
export { StorageBackend }
+// @public
+export class StorageBackendLoadSignal implements LoadSignal {
+ constructor(options?: StorageBackendLoadSignalOptions);
+ // (undocumented)
+ getSample(sampleDurationMillis?: number): LoadSnapshot[];
+ // (undocumented)
+ readonly name = "storageBackendInfo";
+ // (undocumented)
+ readonly overloadedRatio: number;
+ // (undocumented)
+ start(context: LoadSignalStartContext): Promise<void>;
+ // (undocumented)
+ stop(): Promise<void>;
+}
+
+// @public
+export interface StorageBackendLoadSignalOptions {
+ maxErrors?: number;
+ overloadedRatio?: number;
+ snapshotIntervalSecs?: number;
+}
+
export { StorageIdentifier }
// @public
@@ -2140,19 +2140,19 @@ export function supportsDomainThrottling(manager: unknown): manager is SupportsD
// @public
export interface SystemInfo {
// (undocumented)
- clientInfo: ClientInfo;
- // (undocumented)
- cpuInfo: ClientInfo;
+ cpuInfo: LoadSignalInfo;
// (undocumented)
- eventLoopInfo: ClientInfo;
+ eventLoopInfo: LoadSignalInfo;
isSystemIdle: boolean;
- loadSignalInfo?: Record<string, ClientInfo>;
+ loadSignalInfo?: Record<string, LoadSignalInfo>;
// (undocumented)
memCurrentBytes?: number;
// (undocumented)
- memInfo: ClientInfo;
+ memInfo: LoadSignalInfo;
// (undocumented)
memTotalBytes?: number;
+ // (undocumented)
+ storageBackendInfo: LoadSignalInfo;
}
// @public
diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.md
index 7af8ef332..af1d1eefe 100644
--- a/docs/upgrading/upgrading_v4.md
+++ b/docs/upgrading/upgrading_v4.md
@@ -1869,7 +1869,7 @@ new ConcurrencySystem({
memory: { maxUsedRatio: 0.8, overloadedRatio: 0.2 },
eventLoop: { snapshotIntervalSecs: 2, maxBlockedMillis: 100, overloadedRatio: 0.7 },
cpu: { overloadedRatio: 0.4 },
- client: { snapshotIntervalSecs: 1, maxErrors: 3, overloadedRatio: 0.3 },
+ storageBackend: { snapshotIntervalSecs: 1, maxErrors: 3, overloadedRatio: 0.3 },
custom: [myProxyHealthSignal],
},
// the two evaluation windows are policy, alongside the scaling options
@@ -1878,11 +1878,13 @@ new ConcurrencySystem({
});
```
+The signal that watches storage rate-limit errors follows the [`StorageClient` → `StorageBackend` rename](#storagebackend-interface-simplified): the option is `loadSignals.storageBackend`, the class `StorageBackendLoadSignal`, and its verdict is reported as `SystemInfo.storageBackendInfo`. The type of that verdict, `ClientInfo`, is now `LoadSignalInfo` — it backs every signal's entry, not just this one.
+
In detail: the four `max*OverloadedRatio` options of `SystemStatusOptions` were **removed** (each signal now owns its overload ratio, set in its own bag), custom signals moved from `systemStatusOptions.loadSignals` to `loadSignals.custom`, and the two evaluation windows — `snapshotHistorySecs` (autoscaling) and `currentHistorySecs` (task gating) — are plain options on `ConcurrencySystemOptions`, since they apply to every signal alike rather than to any one of them.
-Two related capabilities are new, and covered in the [scaling guide](../guides/scaling-crawlers#load-signals): a built-in signal can be switched **off** with `false`, and each built-in is also a public class (`MemoryLoadSignal`, `EventLoopLoadSignal`, `CpuLoadSignal`, `ClientLoadSignal`) you can construct to wrap or adapt.
+Two related capabilities are new, and covered in the [scaling guide](../guides/scaling-crawlers#load-signals): a built-in signal can be switched **off** with `false`, and each built-in is also a public class (`MemoryLoadSignal`, `EventLoopLoadSignal`, `CpuLoadSignal`, `StorageBackendLoadSignal`) you can construct to wrap or adapt.
-A duplicate signal name now **throws**, where naming a custom signal after a built-in (`memInfo`, `eventLoopInfo`, `cpuInfo`, `clientInfo`) used to look like an override but never was one: the built-in kept running and kept holding concurrency down, while your signal only overwrote its field in the reported `SystemInfo`. To take a built-in's place, switch it off:
+A duplicate signal name now **throws**, where naming a custom signal after a built-in (`memInfo`, `eventLoopInfo`, `cpuInfo`, `storageBackendInfo`) used to look like an override but never was one: the built-in kept running and kept holding concurrency down, while your signal only overwrote its field in the reported `SystemInfo`. To take a built-in's place, switch it off:
```typescript
new ConcurrencySystem({
diff --git a/packages/core/src/autoscaling/concurrency_system.ts b/packages/core/src/autoscaling/concurrency_system.ts
index b50228381..2fdc39776 100644
--- a/packages/core/src/autoscaling/concurrency_system.ts
+++ b/packages/core/src/autoscaling/concurrency_system.ts
@@ -99,8 +99,8 @@ export interface ConcurrencySystemOptions {
/**
* The signals that tell the system whether the machine is overloaded: per-resource tuning for the built-in four
- * (memory, event loop, CPU, client) plus any {@apilink LoadSignalsOptions.custom|`custom`} implementations of
- * your own. See {@apilink LoadSignalsOptions}.
+ * (memory, event loop, CPU, storage backend) plus any {@apilink LoadSignalsOptions.custom|`custom`}
+ * implementations of your own. See {@apilink LoadSignalsOptions}.
*/
loadSignals?: LoadSignalsOptions;
diff --git a/packages/core/src/autoscaling/index.ts b/packages/core/src/autoscaling/index.ts
index 356788ffa..e7d5ed1c5 100644
--- a/packages/core/src/autoscaling/index.ts
+++ b/packages/core/src/autoscaling/index.ts
@@ -1,9 +1,9 @@
export * from './autoscaled_pool.js';
export * from './concurrency_system.js';
-export * from './client_load_signal.js';
export * from './cpu_load_signal.js';
export * from './event_loop_load_signal.js';
export * from './load_signal.js';
export * from './memory_load_signal.js';
export * from './snapshotter.js';
+export * from './storage_backend_load_signal.js';
export * from './system_status.js';
diff --git a/packages/core/src/autoscaling/load_signal.ts b/packages/core/src/autoscaling/load_signal.ts
index d32a69354..0bae4b259 100644
--- a/packages/core/src/autoscaling/load_signal.ts
+++ b/packages/core/src/autoscaling/load_signal.ts
@@ -1,4 +1,4 @@
-import type { ClientInfo } from './system_status.js';
+import type { LoadSignalInfo } from './system_status.js';
import { weightedAvg } from './weighted_avg.js';
/**
@@ -26,7 +26,7 @@ export interface LoadSignalStartContext {
* A signal that reports whether a particular resource is overloaded. The {@apilink ConcurrencySystem} aggregates
* several of them — if any one reports overload, the system is overloaded.
*
- * The built-in signals cover memory, CPU, event loop and storage-client rate limits. Implement this interface to add
+ * The built-in signals cover memory, CPU, event loop and storage backend rate limits. Implement this interface to add
* your own (navigation timeouts, proxy health, …) and pass them via
* {@apilink LoadSignalsOptions.custom|`loadSignals.custom`}; {@apilink SnapshotStore} does the time-windowed
* bookkeeping if you want it. Each built-in is also a public class, so one can be *wrapped* rather than reimplemented
@@ -36,8 +36,8 @@ export interface LoadSignal {
/**
* This signal's key in the reported {@apilink SystemInfo}, also used in logging — so it must be unique among the
* signals of one {@apilink ConcurrencySystem}, which throws on a duplicate. The four built-in names (`memInfo`,
- * `eventLoopInfo`, `cpuInfo`, `clientInfo`) land in the correspondingly named `SystemInfo` fields rather than the
- * `loadSignalInfo` bag; taking one over means switching that built-in off.
+ * `eventLoopInfo`, `cpuInfo`, `storageBackendInfo`) land in the correspondingly named `SystemInfo` fields rather
+ * than the `loadSignalInfo` bag; taking one over means switching that built-in off.
*/
readonly name: string;
@@ -125,7 +125,8 @@ export class SnapshotStore<T extends LoadSnapshot = LoadSnapshot> {
/**
* Direct, unwindowed access to the underlying array — used by signals whose handler needs the previous snapshot
- * to compute a delta (e.g. the event loop and client signals read the last entry to measure change since it).
+ * to compute a delta (e.g. the event loop and storage backend signals read the last entry to measure change since
+ * it).
*/
getAll(): T[] {
return this.#snapshots;
@@ -148,7 +149,7 @@ export class SnapshotStore<T extends LoadSnapshot = LoadSnapshot> {
* evaluation logic used by `SystemStatus` for all signal types.
* @internal
*/
-export function evaluateLoadSignalSample(sample: LoadSnapshot[], overloadedRatio: number): ClientInfo {
+export function evaluateLoadSignalSample(sample: LoadSnapshot[], overloadedRatio: number): LoadSignalInfo {
if (sample.length === 0) {
return {
isOverloaded: false,
diff --git a/packages/core/src/autoscaling/snapshotter.ts b/packages/core/src/autoscaling/snapshotter.ts
index 167fcb27d..c286428ea 100644
--- a/packages/core/src/autoscaling/snapshotter.ts
+++ b/packages/core/src/autoscaling/snapshotter.ts
@@ -1,5 +1,3 @@
-import type { ClientLoadSignalOptions } from './client_load_signal.js';
-import { ClientLoadSignal } from './client_load_signal.js';
import type { CpuLoadSignalOptions } from './cpu_load_signal.js';
import { CpuLoadSignal } from './cpu_load_signal.js';
import type { EventLoopLoadSignalOptions } from './event_loop_load_signal.js';
@@ -7,6 +5,8 @@ import { EventLoopLoadSignal } from './event_loop_load_signal.js';
import type { LoadSignal, LoadSignalStartContext } from './load_signal.js';
import type { MemoryLoadSignalOptions } from './memory_load_signal.js';
import { MemoryLoadSignal } from './memory_load_signal.js';
+import type { StorageBackendLoadSignalOptions } from './storage_backend_load_signal.js';
+import { StorageBackendLoadSignal } from './storage_backend_load_signal.js';
/**
* The load signals a {@apilink ConcurrencySystem} watches to decide whether the machine is overloaded.
@@ -39,11 +39,11 @@ export interface LoadSignalsOptions {
cpu?: CpuLoadSignalOptions | false;
/**
- * Tuning for the built-in {@apilink ClientLoadSignal} (snapshot interval + error limit + overload ratio), or
- * `false` to switch it off — worth doing when the storage backend reports no rate-limit statistics, since the
+ * Tuning for the built-in {@apilink StorageBackendLoadSignal} (snapshot interval + error limit + overload ratio),
+ * or `false` to switch it off — worth doing when the storage backend reports no rate-limit statistics, since the
* signal otherwise polls it every second to no purpose.
*/
- client?: ClientLoadSignalOptions | false;
+ storageBackend?: StorageBackendLoadSignalOptions | false;
/**
* Additional {@apilink LoadSignal} implementations — e.g. navigation timeouts or proxy health — evaluated
@@ -64,19 +64,19 @@ export type SnapshotterOptions = Omit<LoadSignalsOptions, 'custom'>;
/**
* Owns the four built-in {@apilink LoadSignal} instances — {@apilink MemoryLoadSignal},
- * {@apilink EventLoopLoadSignal}, {@apilink CpuLoadSignal} and {@apilink ClientLoadSignal} — constructing the ones
- * that were not switched off and driving their shared lifecycle.
+ * {@apilink EventLoopLoadSignal}, {@apilink CpuLoadSignal} and {@apilink StorageBackendLoadSignal} — constructing
+ * the ones that were not switched off and driving their shared lifecycle.
*
* Configured indirectly through {@apilink ConcurrencySystemOptions.loadSignals|`loadSignals`}, whose per-signal bags
* are simply forwarded to the corresponding constructor.
* @internal
*/
export class Snapshotter {
- // Absent when switched off through the corresponding option (e.g. `client: false`).
+ // Absent when switched off through the corresponding option (e.g. `storageBackend: false`).
readonly #memorySignal?: MemoryLoadSignal;
readonly #eventLoopSignal?: EventLoopLoadSignal;
readonly #cpuSignal?: CpuLoadSignal;
- readonly #clientSignal?: ClientLoadSignal;
+ readonly #storageBackendSignal?: StorageBackendLoadSignal;
/**
* Returns the enabled built-in signals, so `SystemStatus` can iterate them alongside any custom `LoadSignal`
@@ -88,7 +88,7 @@ export class Snapshotter {
this.#memorySignal,
this.#eventLoopSignal,
this.#cpuSignal,
- this.#clientSignal,
+ this.#storageBackendSignal,
];
return builtin.filter((signal): signal is LoadSignal => signal !== undefined);
@@ -98,14 +98,14 @@ export class Snapshotter {
* @param [options] All `Snapshotter` configuration options.
*/
constructor(options: SnapshotterOptions = {}) {
- const { memory = {}, eventLoop = {}, cpu = {}, client = {} } = options;
+ const { memory = {}, eventLoop = {}, cpu = {}, storageBackend = {} } = options;
// Each signal resolves its own ambient dependencies when started, and is told the window it will be sampled
// over then too - so there is nothing to thread in here beyond the caller's tuning.
if (memory !== false) this.#memorySignal = new MemoryLoadSignal(memory);
if (eventLoop !== false) this.#eventLoopSignal = new EventLoopLoadSignal(eventLoop);
if (cpu !== false) this.#cpuSignal = new CpuLoadSignal(cpu);
- if (client !== false) this.#clientSignal = new ClientLoadSignal(client);
+ if (storageBackend !== false) this.#storageBackendSignal = new StorageBackendLoadSignal(storageBackend);
}
/**
diff --git a/packages/core/src/autoscaling/client_load_signal.ts b/packages/core/src/autoscaling/storage_backend_load_signal.ts
similarity index 69%
rename from packages/core/src/autoscaling/client_load_signal.ts
rename to packages/core/src/autoscaling/storage_backend_load_signal.ts
index c30d028bd..1d759af7e 100644
--- a/packages/core/src/autoscaling/client_load_signal.ts
+++ b/packages/core/src/autoscaling/storage_backend_load_signal.ts
@@ -7,23 +7,24 @@ import { serviceLocator } from '../service_locator.js';
import type { LoadSignal, LoadSignalStartContext, LoadSnapshot } from './load_signal.js';
import { SnapshotStore } from './load_signal.js';
-const CLIENT_RATE_LIMIT_ERROR_RETRY_COUNT = 2;
+const RATE_LIMIT_ERROR_RETRY_COUNT = 2;
/**
- * A snapshot produced by the built-in client (rate-limit) signal.
+ * A snapshot produced by the built-in storage backend (rate-limit) signal.
* @internal
*/
-export interface ClientSnapshot extends LoadSnapshot {
+export interface StorageBackendSnapshot extends LoadSnapshot {
rateLimitErrorCount: number;
}
/**
- * Tuning for the built-in **client** (rate-limit) load signal, as accepted both by {@apilink ClientLoadSignal} and by
- * the {@apilink LoadSignalsOptions.client|`client`} shorthand on {@apilink LoadSignalsOptions}.
+ * Tuning for the built-in **storage backend** (rate-limit) load signal, as accepted both by
+ * {@apilink StorageBackendLoadSignal} and by the
+ * {@apilink LoadSignalsOptions.storageBackend|`storageBackend`} shorthand on {@apilink LoadSignalsOptions}.
*/
-export interface ClientLoadSignalOptions {
+export interface StorageBackendLoadSignalOptions {
/**
- * Defines the interval of checking the current state of the remote API client, in seconds.
+ * Defines the interval of checking the current state of the storage backend, in seconds.
* @default 1
*/
snapshotIntervalSecs?: number;
@@ -35,7 +36,7 @@ export interface ClientLoadSignalOptions {
maxErrors?: number;
/**
- * Maximum ratio of overloaded snapshots in a sample before the client counts as overloaded.
+ * Maximum ratio of overloaded snapshots in a sample before the storage backend counts as overloaded.
* @default 0.3
*/
overloadedRatio?: number;
@@ -47,22 +48,22 @@ export interface ClientLoadSignalOptions {
*
* Built by default; construct one yourself only to wrap or adapt it — see {@apilink LoadSignal}.
*
- * Switch it off entirely ({@apilink LoadSignalsOptions.client|`client: false`}) if the storage backend reports no
- * rate-limit statistics, since it otherwise polls it every second to no purpose.
+ * Switch it off entirely ({@apilink LoadSignalsOptions.storageBackend|`storageBackend: false`}) if the storage backend
+ * reports no rate-limit statistics, since it otherwise polls it every second to no purpose.
*
* @category Scaling
*/
-export class ClientLoadSignal implements LoadSignal {
- readonly name = 'clientInfo';
+export class StorageBackendLoadSignal implements LoadSignal {
+ readonly name = 'storageBackendInfo';
readonly overloadedRatio: number;
- readonly #store = new SnapshotStore<ClientSnapshot>();
+ readonly #store = new SnapshotStore<StorageBackendSnapshot>();
readonly #intervalMillis: number;
readonly #maxErrors: number;
#interval?: BetterIntervalID;
- #client?: StorageBackend;
+ #storageBackend?: StorageBackend;
- constructor(options: ClientLoadSignalOptions = {}) {
+ constructor(options: StorageBackendLoadSignalOptions = {}) {
this.overloadedRatio = options.overloadedRatio ?? 0.3;
this.#intervalMillis = (options.snapshotIntervalSecs ?? 1) * 1000;
this.#maxErrors = options.maxErrors ?? 3;
@@ -72,19 +73,19 @@ export class ClientLoadSignal implements LoadSignal {
async start(context: LoadSignalStartContext): Promise<void> {
this.#store.useSampleWindow(context.maxSampleWindowMillis);
// A new session starts from a clean slate, or its first measurement diffs the error count against the previous
- // session's — possibly against a different backend, since the client is resolved afresh just below.
+ // session's — possibly against a different backend, since it is resolved afresh just below.
this.#store.clear();
// Resolved here rather than in the constructor, where asking for the backend would instantiate a default one
// as a side effect - long before the crawler that owns the run has had a chance to register its own.
- this.#client = serviceLocator.getStorageBackend();
+ this.#storageBackend = serviceLocator.getStorageBackend();
this.#interval = betterSetInterval(this.handle, this.#intervalMillis);
}
async stop(): Promise<void> {
if (this.#interval) betterClearInterval(this.#interval);
this.#interval = undefined;
- this.#client = undefined;
+ this.#storageBackend = undefined;
}
getSample(sampleDurationMillis?: number): LoadSnapshot[] {
@@ -99,10 +100,10 @@ export class ClientLoadSignal implements LoadSignal {
handle(intervalCallback: () => unknown): void {
const now = new Date();
- const allErrorCounts = this.#client?.stats?.rateLimitErrors ?? [];
- const currentErrCount = allErrorCounts[CLIENT_RATE_LIMIT_ERROR_RETRY_COUNT] || 0;
+ const allErrorCounts = this.#storageBackend?.stats?.rateLimitErrors ?? [];
+ const currentErrCount = allErrorCounts[RATE_LIMIT_ERROR_RETRY_COUNT] || 0;
- const snapshot: ClientSnapshot = {
+ const snapshot: StorageBackendSnapshot = {
createdAt: now,
isOverloaded: false,
rateLimitErrorCount: currentErrCount,
diff --git a/packages/core/src/autoscaling/system_status.ts b/packages/core/src/autoscaling/system_status.ts
index 37157b2cf..899326d6a 100644
--- a/packages/core/src/autoscaling/system_status.ts
+++ b/packages/core/src/autoscaling/system_status.ts
@@ -8,10 +8,10 @@ import type { LoadSignalsOptions, Snapshotter } from './snapshotter.js';
export interface SystemInfo {
/** If false, system is being overloaded. */
isSystemIdle: boolean;
- memInfo: ClientInfo;
- eventLoopInfo: ClientInfo;
- cpuInfo: ClientInfo;
- clientInfo: ClientInfo;
+ memInfo: LoadSignalInfo;
+ eventLoopInfo: LoadSignalInfo;
+ cpuInfo: LoadSignalInfo;
+ storageBackendInfo: LoadSignalInfo;
memTotalBytes?: number;
memCurrentBytes?: number;
/**
@@ -34,7 +34,7 @@ export interface SystemInfo {
* Status of additional load signals beyond the built-in four.
* Keys are `LoadSignal.name` values, values are overload info.
*/
- loadSignalInfo?: Record<string, ClientInfo>;
+ loadSignalInfo?: Record<string, LoadSignalInfo>;
}
/**
@@ -78,13 +78,13 @@ export interface SystemStatusOptions {
/**
* Additional load signals to include in the system status evaluation.
* These are evaluated alongside the built-in memory, CPU, event loop,
- * and client signals. If any signal reports overload, the system is
- * considered overloaded. Each signal carries its own overload ratio.
+ * and storage backend signals. If any signal reports overload, the system
+ * is considered overloaded. Each signal carries its own overload ratio.
*/
loadSignals?: LoadSignal[];
}
-export interface ClientInfo {
+export interface LoadSignalInfo {
isOverloaded: boolean;
limitRatio: number;
actualRatio: number;
@@ -108,7 +108,7 @@ const BUILTIN_SIGNAL_OPTION_KEYS: Record<string, keyof LoadSignalsOptions> = {
memInfo: 'memory',
eventLoopInfo: 'eventLoop',
cpuInfo: 'cpu',
- clientInfo: 'client',
+ storageBackendInfo: 'storageBackend',
};
const BUILTIN_SIGNAL_NAMES = new Set(Object.keys(BUILTIN_SIGNAL_OPTION_KEYS));
@@ -225,10 +225,10 @@ export class SystemStatus {
memInfo: { isOverloaded: false, limitRatio: 0, actualRatio: 0 },
eventLoopInfo: { isOverloaded: false, limitRatio: 0, actualRatio: 0 },
cpuInfo: { isOverloaded: false, limitRatio: 0, actualRatio: 0 },
- clientInfo: { isOverloaded: false, limitRatio: 0, actualRatio: 0 },
+ storageBackendInfo: { isOverloaded: false, limitRatio: 0, actualRatio: 0 },
};
- let loadSignalInfo: Record<string, ClientInfo> | undefined;
+ let loadSignalInfo: Record<string, LoadSignalInfo> | undefined;
for (const signal of this.#signals) {
const sample = signal.getSample(sampleDurationMillis);
diff --git a/test/core/autoscaling/concurrency_system.test.ts b/test/core/autoscaling/concurrency_system.test.ts
index 313f0af92..0542c973a 100644
--- a/test/core/autoscaling/concurrency_system.test.ts
+++ b/test/core/autoscaling/concurrency_system.test.ts
@@ -191,19 +191,19 @@ describe('ConcurrencySystem', () => {
// The other three built-ins are untouched — each still reports under its own default ratio.
expect(status.eventLoopInfo.limitRatio).toBe(0.6);
expect(status.cpuInfo.limitRatio).toBe(0.4);
- expect(status.clientInfo.limitRatio).toBe(0.3);
+ expect(status.storageBackendInfo.limitRatio).toBe(0.3);
});
test('built-in signals can be switched off with `false`', async () => {
const system = new ConcurrencySystem({
- loadSignals: { client: false, eventLoop: false },
+ loadSignals: { storageBackend: false, eventLoop: false },
});
await system.start();
try {
// A disabled signal cannot report overload, so the status shows it as idle rather than omitting it.
const status = system.getCurrentStatus();
- expect(status.clientInfo).toEqual({ isOverloaded: false, limitRatio: 0, actualRatio: 0 });
+ expect(status.storageBackendInfo).toEqual({ isOverloaded: false, limitRatio: 0, actualRatio: 0 });
expect(status.eventLoopInfo).toEqual({ isOverloaded: false, limitRatio: 0, actualRatio: 0 });
expect(status.isSystemIdle).toBe(true);
@@ -218,7 +218,7 @@ describe('ConcurrencySystem', () => {
test('switching every built-in signal off leaves the system permanently idle', async () => {
const system = new ConcurrencySystem({
- loadSignals: { memory: false, eventLoop: false, cpu: false, client: false },
+ loadSignals: { memory: false, eventLoop: false, cpu: false, storageBackend: false },
});
// @ts-expect-error Accessing private prop
diff --git a/test/core/autoscaling/snapshotter.test.ts b/test/core/autoscaling/snapshotter.test.ts
index d74be4dfb..a8cc45882 100644
--- a/test/core/autoscaling/snapshotter.test.ts
+++ b/test/core/autoscaling/snapshotter.test.ts
@@ -1,7 +1,6 @@
import os from 'node:os';
import {
- ClientLoadSignal,
Configuration,
type EventLoopLoadSignal,
EventType,
@@ -9,6 +8,7 @@ import {
MemoryLoadSignal,
serviceLocator,
Snapshotter,
+ StorageBackendLoadSignal,
} from '@crawlee/core';
import type { MemoryInfo } from '../../../packages/core/src/system-info/memory-info.js';
import * as utils from '../../../packages/core/src/system-info/memory-info.js';
@@ -46,18 +46,18 @@ describe('Snapshotter', () => {
test('should collect snapshots with some values', async () => {
serviceLocator.setConfiguration(new Configuration({ systemInfoIntervalMillis: 100 }));
- // mock client data
- const apifyClient = serviceLocator.getStorageBackend();
- const oldStats = apifyClient.stats;
- apifyClient.stats = {} as any;
- apifyClient.stats!.rateLimitErrors = [0, 0, 0];
+ // mock storage backend data
+ const storageBackend = serviceLocator.getStorageBackend();
+ const oldStats = storageBackend.stats;
+ storageBackend.stats = {} as any;
+ storageBackend.stats!.rateLimitErrors = [0, 0, 0];
const snapshotter = new Snapshotter();
const events = serviceLocator.getEventManager();
await events.init();
await snapshotter.start(START_CONTEXT);
await sleep(625);
- apifyClient.stats!.rateLimitErrors = [0, 0, 2];
+ storageBackend.stats!.rateLimitErrors = [0, 0, 2];
await sleep(625);
await snapshotter.stop();
@@ -65,7 +65,7 @@ describe('Snapshotter', () => {
const memorySnapshots = sampleOf(snapshotter, 'memInfo');
const eventLoopSnapshots = sampleOf(snapshotter, 'eventLoopInfo');
const cpuSnapshots = sampleOf(snapshotter, 'cpuInfo');
- const clientSnapshots = sampleOf(snapshotter, 'clientInfo');
+ const storageBackendSnapshots = sampleOf(snapshotter, 'storageBackendInfo');
expect(Array.isArray(cpuSnapshots)).toBe(true);
expect(cpuSnapshots.length).toBeGreaterThanOrEqual(1);
@@ -91,15 +91,15 @@ describe('Snapshotter', () => {
expect(typeof ss.exceededMillis).toBe('number');
});
- expect(Array.isArray(clientSnapshots)).toBe(true);
- expect(clientSnapshots.length).toBeGreaterThanOrEqual(1);
- clientSnapshots.forEach((ss) => {
+ expect(Array.isArray(storageBackendSnapshots)).toBe(true);
+ expect(storageBackendSnapshots.length).toBeGreaterThanOrEqual(1);
+ storageBackendSnapshots.forEach((ss) => {
expect(ss.createdAt).toBeInstanceOf(Date);
expect(typeof ss.isOverloaded).toBe('boolean');
expect(typeof ss.rateLimitErrorCount).toBe('number');
});
- apifyClient.stats = oldStats;
+ storageBackend.stats = oldStats;
});
test('should override default timers', async () => {
@@ -313,35 +313,35 @@ describe('Snapshotter', () => {
await memorySignal.stop();
});
- test('correctly marks clientOverloaded', async () => {
- // mock client data
- const apifyClient = serviceLocator.getStorageBackend();
- const oldStats = apifyClient.stats;
- apifyClient.stats = {} as any;
- apifyClient.stats!.rateLimitErrors = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
-
- const clientSignal = new ClientLoadSignal({ maxErrors: 1 });
- await clientSignal.start(START_CONTEXT);
- clientSignal.handle(noop);
- apifyClient.stats!.rateLimitErrors = [1, 1, 1, 0, 0, 0, 0, 0, 0, 0];
- clientSignal.handle(noop);
- apifyClient.stats!.rateLimitErrors = [10, 5, 2, 0, 0, 0, 0, 0, 0, 0];
- clientSignal.handle(noop);
- apifyClient.stats!.rateLimitErrors = [100, 24, 4, 2, 0, 0, 0, 0, 0, 0];
- clientSignal.handle(noop);
+ test('correctly marks storageBackendOverloaded', async () => {
+ // mock storage backend data
+ const storageBackend = serviceLocator.getStorageBackend();
+ const oldStats = storageBackend.stats;
+ storageBackend.stats = {} as any;
+ storageBackend.stats!.rateLimitErrors = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
+
+ const storageBackendSignal = new StorageBackendLoadSignal({ maxErrors: 1 });
+ await storageBackendSignal.start(START_CONTEXT);
+ storageBackendSignal.handle(noop);
+ storageBackend.stats!.rateLimitErrors = [1, 1, 1, 0, 0, 0, 0, 0, 0, 0];
+ storageBackendSignal.handle(noop);
+ storageBackend.stats!.rateLimitErrors = [10, 5, 2, 0, 0, 0, 0, 0, 0, 0];
+ storageBackendSignal.handle(noop);
+ storageBackend.stats!.rateLimitErrors = [100, 24, 4, 2, 0, 0, 0, 0, 0, 0];
+ storageBackendSignal.handle(noop);
// `start()` takes a baseline snapshot immediately (the measuring interval fires on its first tick), so the
// four driven below follow it.
- const clientSnapshots = clientSignal.getSample().slice(1);
+ const storageBackendSnapshots = storageBackendSignal.getSample().slice(1);
- expect(clientSnapshots.length).toBe(4);
- expect(clientSnapshots[0].isOverloaded).toBe(false);
- expect(clientSnapshots[1].isOverloaded).toBe(false);
- expect(clientSnapshots[2].isOverloaded).toBe(false);
- expect(clientSnapshots[3].isOverloaded).toBe(true);
+ expect(storageBackendSnapshots.length).toBe(4);
+ expect(storageBackendSnapshots[0].isOverloaded).toBe(false);
+ expect(storageBackendSnapshots[1].isOverloaded).toBe(false);
+ expect(storageBackendSnapshots[2].isOverloaded).toBe(false);
+ expect(storageBackendSnapshots[3].isOverloaded).toBe(true);
- await clientSignal.stop();
- apifyClient.stats = oldStats;
+ await storageBackendSignal.stop();
+ storageBackend.stats = oldStats;
});
test('a signal sample is limited by the requested duration', async () => {
diff --git a/test/core/autoscaling/system_status.test.ts b/test/core/autoscaling/system_status.test.ts
index 626fddf81..7bb1e09bf 100644
--- a/test/core/autoscaling/system_status.test.ts
+++ b/test/core/autoscaling/system_status.test.ts
@@ -33,9 +33,9 @@ describe('SystemStatus', () => {
readonly memSnapshots: any[],
readonly loopSnapshots: any[],
readonly cpuSnapshots: any[],
- readonly clientSnapshots: any[],
+ readonly storageBackendSnapshots: any[],
readonly overloadedRatios: Partial<
- Record<'memInfo' | 'eventLoopInfo' | 'cpuInfo' | 'clientInfo', number>
+ Record<'memInfo' | 'eventLoopInfo' | 'cpuInfo' | 'storageBackendInfo', number>
> = {},
) {}
@@ -44,7 +44,11 @@ describe('SystemStatus', () => {
mockSignal('memInfo', this.memSnapshots, this.overloadedRatios.memInfo),
mockSignal('eventLoopInfo', this.loopSnapshots, this.overloadedRatios.eventLoopInfo),
mockSignal('cpuInfo', this.cpuSnapshots, this.overloadedRatios.cpuInfo),
- mockSignal('clientInfo', this.clientSnapshots, this.overloadedRatios.clientInfo),
+ mockSignal(
+ 'storageBackendInfo',
+ this.storageBackendSnapshots,
+ this.overloadedRatios.storageBackendInfo,
+ ),
];
}
@@ -60,8 +64,8 @@ describe('SystemStatus', () => {
return this.cpuSnapshots.slice(-offset);
}
- getClientSample(offset: number) {
- return this.clientSnapshots.slice(-offset);
+ getStorageBackendSample(offset: number) {
+ return this.storageBackendSnapshots.slice(-offset);
}
}
@@ -146,7 +150,7 @@ describe('SystemStatus', () => {
test('should overload when threshold is crossed', () => {
const snaps = generateSnapsSync(50, true);
- const ratios = (r: number) => ({ memInfo: r, eventLoopInfo: r, cpuInfo: r, clientInfo: r });
+ const ratios = (r: number) => ({ memInfo: r, eventLoopInfo: r, cpuInfo: r, storageBackendInfo: r });
// At exactly 0.5, the 50% overloaded sample should NOT trigger (uses >)
let systemStatus = new SystemStatus({
@@ -162,13 +166,13 @@ describe('SystemStatus', () => {
expect(systemStatus.getCurrentStatus().isSystemIdle).toBe(false);
expect(systemStatus.getHistoricalStatus().isSystemIdle).toBe(false);
- // Memory & eventLoop at threshold, CPU & client below → still overloaded
+ // Memory & eventLoop at threshold, CPU & storage backend below → still overloaded
systemStatus = new SystemStatus({
snapshotter: new MockSnapshotter(snaps, snaps, snaps, snaps, {
memInfo: 0.5,
eventLoopInfo: 0.5,
cpuInfo: 0.49,
- clientInfo: 0.49,
+ storageBackendInfo: 0.49,
}) as any,
});
expect(systemStatus.getCurrentStatus().isSystemIdle).toBe(false);
@@ -183,7 +187,7 @@ describe('SystemStatus', () => {
});
test('should show different values for now and lately', () => {
- const ratios = { memInfo: 0.5, eventLoopInfo: 0.5, cpuInfo: 0.5, clientInfo: 0.5 };
+ const ratios = { memInfo: 0.5, eventLoopInfo: 0.5, cpuInfo: 0.5, storageBackendInfo: 0.5 };
// The "now" window is configuration, so each width gets its own instance rather than a mutated one.
const statusWithCurrentWindow = (snaps: any[], currentHistoryMillis: number) =>
new SystemStatus({