Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

feat: Pass task identifiers from AutoscaledPool to IConcurrencySystem (#4130)

Jan Buchar7eb08812271e
diff --git a/docs/guides/scaling_crawlers.mdx b/docs/guides/scaling_crawlers.mdx
index dbe42c49f..ecc9fabe5 100644
--- a/docs/guides/scaling_crawlers.mdx
+++ b/docs/guides/scaling_crawlers.mdx
@@ -14,6 +14,7 @@ import ConcurrencySystemSource from '!!raw-loader!./scaling_crawlers_concurrency
import LoadSignalsOffSource from '!!raw-loader!./scaling_crawlers_loadSignalsOff.ts';
import CustomLoadSignalSource from '!!raw-loader!./scaling_crawlers_customLoadSignal.ts';
import WrapLoadSignalSource from '!!raw-loader!./scaling_crawlers_wrapLoadSignal.ts';
+import CustomConcurrencySystemSource from '!!raw-loader!./scaling_crawlers_customConcurrencySystem.ts';
As we build our crawler, we might want to control how many requests we do to the website at a time. Crawlee provides several options to fine tune how many parallel requests should be made at any time, how many requests should be done per minute, and how should scaling work based on the available system resources.
@@ -72,7 +73,7 @@ The `await using` syntax needs Node.js 24 or later. On Node.js 22 call <ApiLink
Injecting the *same* `ConcurrencySystem` instance into several crawlers makes them share a single concurrency budget, capping their combined parallelism instead of letting each crawler scale independently.
-The budget is shared, not divided: slots go to whoever asks first, so a crawler that fills it early keeps refilling it as its own requests finish, and one that starts later may get a much smaller share of it. If a crawler needs a guaranteed slice, give it its own instance — or implement <ApiLink to="basic-crawler/interface/IConcurrencySystem">`IConcurrencySystem`</ApiLink>, whose allocation methods are told which crawler is asking, and allocate however we see fit.
+The budget is shared, not divided: slots go to whoever asks first, so a crawler that fills it early keeps refilling it as its own requests finish, and one that starts later may get a much smaller share of it. If a crawler needs a guaranteed slice, give it its own instance — or [implement the governor ourselves](#custom-concurrency-systems).
:::
@@ -126,6 +127,16 @@ This option can be set by specifying [`maxRequestsPerMinute`](#maxrequestspermin
:::
+## Custom concurrency systems
+
+`ConcurrencySystem` is just one implementation of <ApiLink to="basic-crawler/interface/IConcurrencySystem">`IConcurrencySystem`</ApiLink>, which is the contract a crawler actually depends on. Writing our own replaces the allocation policy wholesale — here with one that gives every crawler a guaranteed slice, rather than serving whoever asks first:
+
+<CodeBlock language="js">
+ {CustomConcurrencySystemSource}
+</CodeBlock>
+
+Both allocation methods are told which crawler is asking and which of its tasks is being booked or released. The consumer is the same object on every call one crawler makes, and task ids are unique within a crawler and handed out in booking order — so the two together identify a task, and pairing a booking with its release is what lets an implementation time tasks, or tell which ones are still in flight.
+
## Load signals
Whether the machine counts as overloaded is decided by *load signals*. Four are built in — memory, event loop, CPU and the storage backend's rate-limit errors — and each is configured by its own bag under <ApiLink to="basic-crawler/interface/ConcurrencySystemOptions#loadSignals">`loadSignals`</ApiLink>, carrying both its limits and the `overloadedRatio` at which it fires. If any signal reports overload, the system is overloaded and concurrency is held down.
diff --git a/docs/guides/scaling_crawlers_customConcurrencySystem.ts b/docs/guides/scaling_crawlers_customConcurrencySystem.ts
new file mode 100644
index 000000000..c2897c652
--- /dev/null
+++ b/docs/guides/scaling_crawlers_customConcurrencySystem.ts
@@ -0,0 +1,41 @@
+import type { ConcurrencyConsumer, IConcurrencySystem } from 'crawlee';
+import { CheerioCrawler } from 'crawlee';
+
+const SLOTS_PER_CRAWLER = 5;
+
+// Task ids repeat across consumers - each pool numbers its own tasks - so the consumer, which is the
+// same object on every call it makes, is what keeps their bookings apart.
+const inFlight = new Map<ConcurrencyConsumer, Map<string, number>>();
+
+// A governor that hands every crawler the same guaranteed slice, instead of serving whoever asks first.
+const fairShare: IConcurrencySystem = {
+ desiredConcurrency: SLOTS_PER_CRAWLER,
+ isRunning: true,
+ get currentConcurrency() {
+ return [...inFlight.values()].reduce((total, tasks) => total + tasks.size, 0);
+ },
+ hasCapacityForTask: (consumer) => (inFlight.get(consumer)?.size ?? 0) < SLOTS_PER_CRAWLER,
+ tryRegisterTaskStart(consumer, taskId) {
+ const tasks = inFlight.get(consumer) ?? new Map<string, number>();
+ inFlight.set(consumer, tasks);
+
+ if (tasks.size >= SLOTS_PER_CRAWLER) {
+ return false;
+ }
+
+ tasks.set(taskId, Date.now());
+
+ return true;
+ },
+ registerTaskEnd(consumer, taskId) {
+ const tasks = inFlight.get(consumer)!;
+
+ console.log(`${consumer.id} ran task ${taskId} for ${Date.now() - tasks.get(taskId)!} ms`);
+ tasks.delete(taskId);
+ },
+};
+
+const products = new CheerioCrawler({ id: 'products', concurrencySystem: fairShare });
+const reviews = new CheerioCrawler({ id: 'reviews', concurrencySystem: fairShare });
+
+await Promise.all([products.run(['https://crawlee.dev']), reviews.run(['https://crawlee.dev'])]);
diff --git a/docs/public-api/crawlee-basic.api.md b/docs/public-api/crawlee-basic.api.md
index 6814720f8..7dbf18315 100644
--- a/docs/public-api/crawlee-basic.api.md
+++ b/docs/public-api/crawlee-basic.api.md
@@ -522,8 +522,8 @@ export interface IConcurrencySystem {
readonly desiredConcurrency: number;
hasCapacityForTask(consumer: ConcurrencyConsumer): boolean;
readonly isRunning: boolean;
- registerTaskEnd(consumer: ConcurrencyConsumer): void;
- tryRegisterTaskStart(consumer: ConcurrencyConsumer): boolean;
+ registerTaskEnd(consumer: ConcurrencyConsumer, taskId: string): void;
+ tryRegisterTaskStart(consumer: ConcurrencyConsumer, taskId: string): boolean;
}
// @public
diff --git a/packages/basic-crawler/src/internals/autoscaling/autoscaled_pool.ts b/packages/basic-crawler/src/internals/autoscaling/autoscaled_pool.ts
index 83e50b8e4..33263ea97 100644
--- a/packages/basic-crawler/src/internals/autoscaling/autoscaled_pool.ts
+++ b/packages/basic-crawler/src/internals/autoscaling/autoscaled_pool.ts
@@ -81,8 +81,10 @@ export interface AutoscaledPoolOptions extends TaskLoopOptions {
/**
* A function that performs an asynchronous resource-intensive task.
* The function must either be labeled `async` or return a promise.
+ * @param taskId The id the task is {@apilink IConcurrencySystem.tryRegisterTaskStart|booked} under, so work done
+ * inside can be correlated with what the governor sees.
*/
- runTaskFunction?: () => Promise<unknown>;
+ runTaskFunction?: (taskId: string) => Promise<unknown>;
/**
* Timeout in which the `runTaskFunction` needs to finish, given in seconds.
@@ -151,7 +153,7 @@ export class AutoscaledPool {
// Configurable properties.
readonly #maybeRunIntervalMillis: number;
readonly #taskTimeoutMillis: number;
- readonly #runTaskFunction: () => Promise<unknown>;
+ readonly #runTaskFunction: (taskId: string) => Promise<unknown>;
readonly #isFinishedFunction: () => Promise<boolean>;
readonly #isTaskReadyFunction: () => Promise<boolean>;
@@ -172,6 +174,9 @@ export class AutoscaledPool {
*/
#ownConcurrency = 0;
+ /** Source of the ids that pair a booking with the release of the same task. */
+ #nextTaskId = 0;
+
constructor(options: AutoscaledPoolOptions) {
const {
runTaskFunction,
@@ -403,9 +408,11 @@ export class AutoscaledPool {
return this.maybeFinish();
}
+ const taskId = String(this.#nextTaskId++);
+
// - the budget still has room. Re-checked atomically, because another pool sharing the governor may have taken
// the last free slot while we awaited `isTaskReadyFunction` above.
- if (!this.#concurrencySystem.tryRegisterTaskStart(this.#consumer)) {
+ if (!this.#concurrencySystem.tryRegisterTaskStart(this.#consumer, taskId)) {
return done();
}
@@ -425,12 +432,12 @@ export class AutoscaledPool {
if (this.#taskTimeoutMillis > 0) {
await addTimeoutToPromise(
- async () => this.#runTaskFunction(),
+ async () => this.#runTaskFunction(taskId),
this.#taskTimeoutMillis,
`runTaskFunction timed out after ${this.#taskTimeoutMillis / 1000} seconds.`,
);
} else {
- await this.#runTaskFunction();
+ await this.#runTaskFunction(taskId);
}
this.#log.perf('Task finished.');
@@ -452,7 +459,7 @@ export class AutoscaledPool {
this.#reject(err);
}
} finally {
- this.#concurrencySystem.registerTaskEnd(this.#consumer);
+ this.#concurrencySystem.registerTaskEnd(this.#consumer, taskId);
this.#ownConcurrency--;
}
diff --git a/packages/basic-crawler/src/internals/autoscaling/concurrency_system.ts b/packages/basic-crawler/src/internals/autoscaling/concurrency_system.ts
index 8cedfcb5e..d41352821 100644
--- a/packages/basic-crawler/src/internals/autoscaling/concurrency_system.ts
+++ b/packages/basic-crawler/src/internals/autoscaling/concurrency_system.ts
@@ -187,11 +187,18 @@ export interface IConcurrencySystem {
*
* Must be an *atomic* (synchronous) check-and-book: several pools may share one governor, and a check separated
* from the booking by an `await` lets two of them claim the last free slot at once.
+ * @param taskId Identifies the booked task among the ones `consumer` has in flight, and is repeated by the
+ * {@apilink IConcurrencySystem.registerTaskEnd|release} — pairing the two is what lets an implementation time a
+ * task or tell which ones are still running. Nothing beyond that is promised: another consumer may be using the
+ * same id at the same time, and an id may come back around once its task has ended.
*/
- tryRegisterTaskStart(consumer: ConcurrencyConsumer): boolean;
+ tryRegisterTaskStart(consumer: ConcurrencyConsumer, taskId: string): boolean;
- /** Returns a task's slot to `consumer`'s budget. Called once the task settles (resolve or reject). */
- registerTaskEnd(consumer: ConcurrencyConsumer): void;
+ /**
+ * Returns a task's slot to `consumer`'s budget. Called once the task settles (resolve or reject).
+ * @param taskId The id the task was {@apilink IConcurrencySystem.tryRegisterTaskStart|booked} under.
+ */
+ registerTaskEnd(consumer: ConcurrencyConsumer, taskId: string): void;
}
/**
diff --git a/test/core/autoscaling/autoscaled_pool.test.ts b/test/core/autoscaling/autoscaled_pool.test.ts
index 0ec2540e7..8c653aa65 100644
--- a/test/core/autoscaling/autoscaled_pool.test.ts
+++ b/test/core/autoscaling/autoscaled_pool.test.ts
@@ -282,13 +282,13 @@ describe('AutoscaledPool', () => {
const origStart = pool.system.tryRegisterTaskStart.bind(pool.system);
const origEnd = pool.system.registerTaskEnd.bind(pool.system);
- vitest.spyOn(pool.system, 'tryRegisterTaskStart').mockImplementation((consumer) => {
- const res = origStart(consumer);
+ vitest.spyOn(pool.system, 'tryRegisterTaskStart').mockImplementation((consumer, taskId) => {
+ const res = origStart(consumer, taskId);
concurrencyLog.push(pool.system.currentConcurrency);
return res;
});
- vitest.spyOn(pool.system, 'registerTaskEnd').mockImplementation((consumer) => {
- origEnd(consumer);
+ vitest.spyOn(pool.system, 'registerTaskEnd').mockImplementation((consumer, taskId) => {
+ origEnd(consumer, taskId);
concurrencyLog.push(pool.system.currentConcurrency);
});
diff --git a/test/core/autoscaling/concurrency_system.test.ts b/test/core/autoscaling/concurrency_system.test.ts
index 5dc301a33..f2ffacef3 100644
--- a/test/core/autoscaling/concurrency_system.test.ts
+++ b/test/core/autoscaling/concurrency_system.test.ts
@@ -686,6 +686,60 @@ describe('ConcurrencySystem', () => {
}
});
+ test('a task, its booking and its release all carry the same id', async () => {
+ const inFlight = new Set<string>();
+ const bookings: string[] = [];
+ const unmatchedReleases: string[] = [];
+ const unbookedTasks: string[] = [];
+ let peakInFlight = 0;
+
+ const governor: IConcurrencySystem = {
+ desiredConcurrency: 2,
+ currentConcurrency: 0,
+ isRunning: true,
+ hasCapacityForTask: () => inFlight.size < 2,
+ tryRegisterTaskStart: (_consumer, taskId) => {
+ if (inFlight.size >= 2) return false;
+ bookings.push(taskId);
+ inFlight.add(taskId);
+ peakInFlight = Math.max(peakInFlight, inFlight.size);
+ return true;
+ },
+ registerTaskEnd: (_consumer, taskId) => {
+ if (!inFlight.delete(taskId)) {
+ unmatchedReleases.push(taskId);
+ }
+ },
+ };
+
+ let done = 0;
+ const pool = new AutoscaledPool({
+ concurrencySystem: governor,
+ consumer: { id: 'pool' },
+ runTaskFunction: async (taskId) => {
+ if (!inFlight.has(taskId)) {
+ unbookedTasks.push(taskId);
+ }
+ await sleep(20);
+ done++;
+ },
+ isFinishedFunction: async () => done >= 6,
+ isTaskReadyFunction: async () => done < 6,
+ });
+
+ await pool.run();
+
+ // The point of the ids: every release can be matched to the booking it settles, so a governor can time
+ // its tasks and know which ones are still running.
+ expect(unmatchedReleases).toEqual([]);
+ // The task runs under the id its own slot was booked with, so it can attribute its work to that booking.
+ expect(unbookedTasks).toEqual([]);
+ expect(inFlight.size).toBe(0);
+ // Ids are never recycled while their task runs, or the two concurrent bookings would collide.
+ expect(peakInFlight).toBe(2);
+ expect(new Set(bookings).size).toBe(bookings.length);
+ });
+
test('tuning a shared governor is reflected by every borrowing pool', () => {
const system = new ConcurrencySystem();