Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

refactor: Remove unnecessary retries when enqueueing new requests (#3962)

Jan Bucharf2f43976ac82
diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.md
index b80339cb5..797ba1921 100644
--- a/docs/upgrading/upgrading_v4.md
+++ b/docs/upgrading/upgrading_v4.md
@@ -1119,6 +1119,10 @@ The same change applies to `CrawlingContext.getKeyValueStore()` and `CrawlingCon
The request loader/manager interfaces have been reworked to mirror the abstractions in Crawlee for Python. See the new [Request loaders](../guides/request-loaders) guide for the full picture.
+### `RequestQueue.addRequestsBatched` no longer retries rejected requests
+
+Requests that the storage backend reports as unprocessed are now warned about and skipped after the first attempt, instead of being retried a bounded number of times. What a backend reports as unprocessed is a semantic rejection (typically malformed request data) that re-sending cannot fix — retrying transient failures is the storage backend's own responsibility.
+
### `IRequestList` renamed to `IRequestLoader`
The `IRequestList` interface has been renamed to `IRequestLoader` and is now the read-only base interface implemented by `RequestList` and `SitemapRequestLoader`. The writable `IRequestManager` interface now **extends** `IRequestLoader` with the request-adding and reclaiming surface (`addRequest`, `addRequestsBatched`, `reclaimRequest`, optional `purge`). There is no `IRequestList` alias — update your imports and type references to `IRequestLoader` (or `IRequestManager` if you need the write surface).
diff --git a/packages/core/src/storages/request_queue.ts b/packages/core/src/storages/request_queue.ts
index 2a647fe33..4f611357b 100644
--- a/packages/core/src/storages/request_queue.ts
+++ b/packages/core/src/storages/request_queue.ts
@@ -42,12 +42,6 @@ import { RequestDeduplicationCache } from './request_dedup_cache.js';
*/
const MAX_CACHED_REQUESTS = 2_000_000;
-/**
- * The maximum number of consecutive no-progress retries for unprocessed requests in `addRequestsBatched()`.
- * @internal
- */
-const MAX_UNPROCESSED_REQUESTS_RETRIES = 3;
-
/**
* Represents a queue of URLs to crawl, which is used for deep crawling of websites
* where you start with several URLs and then recursively
@@ -451,56 +445,33 @@ export class RequestQueue implements IStorage, IRequestManager {
);
const chunksIterator = chunks[Symbol.asyncIterator]();
- const attemptToAddToQueueAndAddAnyUnprocessed = async (
- providedRequests: Source[],
- cache = true,
- unsuccessfulAttempts = 0,
- ) => {
- const resultsToReturn: ProcessedRequest[] = [];
- const apiResult = await this.addRequests(providedRequests, { forefront: options.forefront, cache });
- resultsToReturn.push(...apiResult.processedRequests);
-
- if (apiResult.unprocessedRequests.length) {
- // Count attempts that make no progress, so permanently rejected requests (e.g. a malformed
- // `userData` shape causing a 400) don't loop forever. Any progress resets the counter.
- const attempts = apiResult.processedRequests.length ? 0 : unsuccessfulAttempts + 1;
-
- if (attempts >= MAX_UNPROCESSED_REQUESTS_RETRIES) {
- this.log.warning(
- `Some requests were consistently rejected by the request queue and will be skipped after ${MAX_UNPROCESSED_REQUESTS_RETRIES} attempts. This usually means the request data is malformed (e.g. an invalid 'userData' shape).`,
- { unprocessedRequests: apiResult.unprocessedRequests },
- );
-
- return resultsToReturn;
- }
-
- await sleep(waitBetweenBatchesMillis);
-
- resultsToReturn.push(
- ...(await attemptToAddToQueueAndAddAnyUnprocessed(
- providedRequests.filter(
- (r) => !apiResult.processedRequests.some((pr) => pr.uniqueKey === r.uniqueKey),
- ),
- false,
- attempts,
- )),
- );
- }
-
- return resultsToReturn;
- };
-
/**
* Process a chunk: send it to the queue, then update the remaining budget if maxNewRequests is active.
+ *
+ * Requests the backend reports as unprocessed are warned about and skipped rather than retried:
+ * `unprocessedRequests` is what remains after the backend's own transient-error handling - a
+ * semantic rejection (e.g. a malformed `userData` shape) that re-sending would only re-poke.
+ * Retrying transient failures is the storage backend's job, not the frontend's.
*/
const processChunk = async (chunk: Source[], cache = true) => {
- const results = await attemptToAddToQueueAndAddAnyUnprocessed(chunk, cache);
+ const { processedRequests, unprocessedRequests } = await this.addRequests(chunk, {
+ forefront: options.forefront,
+ cache,
+ });
+
+ if (unprocessedRequests.length > 0) {
+ this.log.warning(
+ 'Some requests were rejected by the request queue and will be skipped. ' +
+ "This usually means the request data is malformed (e.g. an invalid 'userData' shape).",
+ { unprocessedRequests },
+ );
+ }
if (maxNewRequests !== undefined) {
- remainingBudget -= results.filter((r) => !r.wasAlreadyPresent).length;
+ remainingBudget -= processedRequests.filter((r) => !r.wasAlreadyPresent).length;
}
- return results;
+ return processedRequests;
};
/**
diff --git a/test/core/storages/request_queue.test.ts b/test/core/storages/request_queue.test.ts
index e01b97858..d2b697aed 100644
--- a/test/core/storages/request_queue.test.ts
+++ b/test/core/storages/request_queue.test.ts
@@ -135,7 +135,7 @@ describe('RequestQueue remote', () => {
expect(fetchedUrls.sort()).toEqual(['http://example.com/a', 'http://example.com/b']);
});
- test('addRequestsBatched does not retry permanently unprocessed requests forever', async () => {
+ test('addRequestsBatched warns about and skips requests the backend rejects, without retrying', async () => {
const queue = await RequestQueue.open();
const mockAddRequests = vitest.spyOn(queue.backend, 'addBatchOfRequests');
@@ -153,10 +153,15 @@ describe('RequestQueue remote', () => {
const result = await queue.addRequestsBatched([requestOptions], { waitBetweenBatchesMillis: 0 });
- // Must not hang: it gives up after a bounded number of attempts and warns about the skipped requests.
+ // Retrying transient failures is the backend's own job: the frontend makes one attempt,
+ // warns about the rejected requests, and moves on.
expect(result.addedRequests).toHaveLength(0);
- expect(logWarningSpy).toHaveBeenCalled();
- expect(mockAddRequests.mock.calls.length).toBeLessThan(20);
+ expect(mockAddRequests).toHaveBeenCalledTimes(1);
+ expect(logWarningSpy).toHaveBeenCalledTimes(1);
+ expect(logWarningSpy.mock.calls[0][0]).toMatch(/rejected by the request queue/);
+ expect(logWarningSpy.mock.calls[0][1]).toMatchObject({
+ unprocessedRequests: [{ uniqueKey: request.uniqueKey, url: request.url, method: 'GET' }],
+ });
});
test('addRequestsBatched does not re-submit already enqueued requests beyond the initial batch (#3120)', async () => {