Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

fix(core): solve possible dead locks in `RequestQueueV2` (#2376)

Vlad Franguffba095c8a74
diff --git a/docs/experiments/request_locking.mdx b/docs/experiments/request_locking.mdx
index 30c9cd74d..f4f00116f 100644
--- a/docs/experiments/request_locking.mdx
+++ b/docs/experiments/request_locking.mdx
@@ -21,7 +21,7 @@ API. With this API, you will be able to pass a `RequestQueue` to multiple crawle
:::info Keep in mind
-The request queue that supports request locking is currently exported via the `RequestQueue2` class. Once the experiment is over, this class will replace
+The request queue that supports request locking is currently exported via the `RequestQueueV2` class. Once the experiment is over, this class will replace
the current `RequestQueue` class
:::
@@ -58,13 +58,13 @@ await crawler.run(['https://crawlee.dev']);
### Outside crawlers (to setup your own request queue that supports locking)
-Previously, you would import `RequestQueue` from `crawlee`. To switch to the queue that supports locking, you need to import `RequestQueue2` instead.
+Previously, you would import `RequestQueue` from `crawlee`. To switch to the queue that supports locking, you need to import `RequestQueueV2` instead.
```ts
// highlight-next-line
-import { RequestQueue2 } from 'crawlee';
+import { RequestQueueV2 } from 'crawlee';
-const queue = await RequestQueue2.open('my-locking-queue');
+const queue = await RequestQueueV2.open('my-locking-queue');
await queue.addRequests([
{ url: 'https://crawlee.dev' },
{ url: 'https://crawlee.dev/docs' },
@@ -83,10 +83,10 @@ If you do not enable the experiment, you will receive a runtime error and the cr
:::
```ts
-import { CheerioCrawler, RequestQueue2 } from 'crawlee';
+import { CheerioCrawler, RequestQueueV2 } from 'crawlee';
// highlight-next-line
-const queue = await RequestQueue2.open('my-locking-queue');
+const queue = await RequestQueueV2.open('my-locking-queue');
const crawler = new CheerioCrawler({
// highlight-next-line
diff --git a/packages/core/src/storages/request_provider.ts b/packages/core/src/storages/request_provider.ts
index 59efeb8a1..6fd85ca91 100644
--- a/packages/core/src/storages/request_provider.ts
+++ b/packages/core/src/storages/request_provider.ts
@@ -16,7 +16,7 @@ import ow from 'ow';
import { checkStorageAccess } from './access_checking';
import type { IStorage, StorageManagerOptions } from './storage_manager';
import { StorageManager } from './storage_manager';
-import { QUERY_HEAD_MIN_LENGTH, STORAGE_CONSISTENCY_DELAY_MILLIS, getRequestId, purgeDefaultStorages } from './utils';
+import { QUERY_HEAD_MIN_LENGTH, getRequestId, purgeDefaultStorages } from './utils';
import { Configuration } from '../configuration';
import { EventType } from '../events';
import { log } from '../log';
@@ -444,20 +444,6 @@ export abstract class RequestProvider implements IStorage {
queueOperationInfo.uniqueKey = request.uniqueKey;
this._cacheRequest(getRequestId(request.uniqueKey), queueOperationInfo);
- // Wait a little to increase a chance that the next call to fetchNextRequest() will return the request with updated data.
- // This is to compensate for the limitation of DynamoDB, where writes might not be immediately visible to subsequent reads.
- setTimeout(() => {
- if (!this.inProgress.has(request.id)) {
- this.log.debug('The request is no longer marked as in progress in the queue?!', { requestId: request.id });
- return;
- }
-
- this.inProgress.delete(request.id);
-
- // Performance optimization: add request straight to head if possible
- this._maybeAddRequestToQueueHead(request.id, forefront);
- }, STORAGE_CONSISTENCY_DELAY_MILLIS);
-
return queueOperationInfo;
}
@@ -500,6 +486,9 @@ export abstract class RequestProvider implements IStorage {
* Caches information about request to beware of unneeded addRequest() calls.
*/
protected _cacheRequest(cacheKey: string, queueOperationInfo: RequestQueueOperationInfo): void {
+ // Remove the previous entry, as otherwise our cache will never update 👀
+ this.requestCache.remove(cacheKey);
+
this.requestCache.add(cacheKey, {
id: queueOperationInfo.requestId,
isHandled: queueOperationInfo.wasAlreadyHandled,
diff --git a/packages/core/src/storages/request_queue.ts b/packages/core/src/storages/request_queue.ts
index d4e6aa35b..ae8ed86a8 100644
--- a/packages/core/src/storages/request_queue.ts
+++ b/packages/core/src/storages/request_queue.ts
@@ -331,7 +331,27 @@ export class RequestQueue extends RequestProvider {
checkStorageAccess();
this.lastActivity = new Date();
- return super.reclaimRequest(...args);
+
+ const [request, options] = args;
+ const forefront = options?.forefront ?? false;
+
+ const result = await super.reclaimRequest(...args);
+
+ // Wait a little to increase a chance that the next call to fetchNextRequest() will return the request with updated data.
+ // This is to compensate for the limitation of DynamoDB, where writes might not be immediately visible to subsequent reads.
+ setTimeout(() => {
+ if (!this.inProgress.has(request.id!)) {
+ this.log.debug('The request is no longer marked as in progress in the queue?!', { requestId: request.id });
+ return;
+ }
+
+ this.inProgress.delete(request.id!);
+
+ // Performance optimization: add request straight to head if possible
+ this._maybeAddRequestToQueueHead(request.id!, forefront);
+ }, STORAGE_CONSISTENCY_DELAY_MILLIS);
+
+ return result;
}
protected override _reset() {
diff --git a/packages/core/src/storages/request_queue_v2.ts b/packages/core/src/storages/request_queue_v2.ts
index 0808bd4a9..58af926b2 100644
--- a/packages/core/src/storages/request_queue_v2.ts
+++ b/packages/core/src/storages/request_queue_v2.ts
@@ -51,6 +51,8 @@ class RequestQueue extends RequestProvider {
super._cacheRequest(cacheKey, queueOperationInfo);
+ this.requestCache.remove(queueOperationInfo.requestId);
+
this.requestCache.add(queueOperationInfo.requestId, {
id: queueOperationInfo.requestId,
isHandled: queueOperationInfo.wasAlreadyHandled,
@@ -149,6 +151,11 @@ class RequestQueue extends RequestProvider {
if (res) {
const [request, options] = args;
+ // Mark the request as no longer in progress,
+ // as the moment we delete the lock, we could end up also re-fetching the request in a subsequent ensureHeadIsNonEmpty()
+ // which could potentially lock the request again
+ this.inProgress.delete(request.id!);
+
// Try to delete the request lock if possible
try {
await this.client.deleteRequestLock(request.id!, { forefront: options?.forefront ?? false });
@@ -188,6 +195,21 @@ class RequestQueue extends RequestProvider {
for (const { id, uniqueKey } of headData.items) {
// Queue head index might be behind the main table, so ensure we don't recycle requests
if (!id || !uniqueKey || this.inProgress.has(id) || this.recentlyHandledRequestsCache.get(id)) {
+ this.log.debug(`Skipping request from queue head, already in progress or recently handled`, {
+ id,
+ uniqueKey,
+ inProgress: this.inProgress.has(id),
+ recentlyHandled: !!this.recentlyHandledRequestsCache.get(id),
+ });
+
+ // Remove the lock from the request for now, so that it can be picked up later
+ // This may/may not succeed, but that's fine
+ try {
+ await this.client.deleteRequestLock(id);
+ } catch {
+ // Ignore
+ }
+
continue;
}
diff --git a/website/versioned_docs/version-3.5/experiments/request_locking.mdx b/website/versioned_docs/version-3.5/experiments/request_locking.mdx
index 30c9cd74d..f4f00116f 100644
--- a/website/versioned_docs/version-3.5/experiments/request_locking.mdx
+++ b/website/versioned_docs/version-3.5/experiments/request_locking.mdx
@@ -21,7 +21,7 @@ API. With this API, you will be able to pass a `RequestQueue` to multiple crawle
:::info Keep in mind
-The request queue that supports request locking is currently exported via the `RequestQueue2` class. Once the experiment is over, this class will replace
+The request queue that supports request locking is currently exported via the `RequestQueueV2` class. Once the experiment is over, this class will replace
the current `RequestQueue` class
:::
@@ -58,13 +58,13 @@ await crawler.run(['https://crawlee.dev']);
### Outside crawlers (to setup your own request queue that supports locking)
-Previously, you would import `RequestQueue` from `crawlee`. To switch to the queue that supports locking, you need to import `RequestQueue2` instead.
+Previously, you would import `RequestQueue` from `crawlee`. To switch to the queue that supports locking, you need to import `RequestQueueV2` instead.
```ts
// highlight-next-line
-import { RequestQueue2 } from 'crawlee';
+import { RequestQueueV2 } from 'crawlee';
-const queue = await RequestQueue2.open('my-locking-queue');
+const queue = await RequestQueueV2.open('my-locking-queue');
await queue.addRequests([
{ url: 'https://crawlee.dev' },
{ url: 'https://crawlee.dev/docs' },
@@ -83,10 +83,10 @@ If you do not enable the experiment, you will receive a runtime error and the cr
:::
```ts
-import { CheerioCrawler, RequestQueue2 } from 'crawlee';
+import { CheerioCrawler, RequestQueueV2 } from 'crawlee';
// highlight-next-line
-const queue = await RequestQueue2.open('my-locking-queue');
+const queue = await RequestQueueV2.open('my-locking-queue');
const crawler = new CheerioCrawler({
// highlight-next-line
diff --git a/website/versioned_docs/version-3.6/experiments/request_locking.mdx b/website/versioned_docs/version-3.6/experiments/request_locking.mdx
index 30c9cd74d..f4f00116f 100644
--- a/website/versioned_docs/version-3.6/experiments/request_locking.mdx
+++ b/website/versioned_docs/version-3.6/experiments/request_locking.mdx
@@ -21,7 +21,7 @@ API. With this API, you will be able to pass a `RequestQueue` to multiple crawle
:::info Keep in mind
-The request queue that supports request locking is currently exported via the `RequestQueue2` class. Once the experiment is over, this class will replace
+The request queue that supports request locking is currently exported via the `RequestQueueV2` class. Once the experiment is over, this class will replace
the current `RequestQueue` class
:::
@@ -58,13 +58,13 @@ await crawler.run(['https://crawlee.dev']);
### Outside crawlers (to setup your own request queue that supports locking)
-Previously, you would import `RequestQueue` from `crawlee`. To switch to the queue that supports locking, you need to import `RequestQueue2` instead.
+Previously, you would import `RequestQueue` from `crawlee`. To switch to the queue that supports locking, you need to import `RequestQueueV2` instead.
```ts
// highlight-next-line
-import { RequestQueue2 } from 'crawlee';
+import { RequestQueueV2 } from 'crawlee';
-const queue = await RequestQueue2.open('my-locking-queue');
+const queue = await RequestQueueV2.open('my-locking-queue');
await queue.addRequests([
{ url: 'https://crawlee.dev' },
{ url: 'https://crawlee.dev/docs' },
@@ -83,10 +83,10 @@ If you do not enable the experiment, you will receive a runtime error and the cr
:::
```ts
-import { CheerioCrawler, RequestQueue2 } from 'crawlee';
+import { CheerioCrawler, RequestQueueV2 } from 'crawlee';
// highlight-next-line
-const queue = await RequestQueue2.open('my-locking-queue');
+const queue = await RequestQueueV2.open('my-locking-queue');
const crawler = new CheerioCrawler({
// highlight-next-line
diff --git a/website/versioned_docs/version-3.7/experiments/request_locking.mdx b/website/versioned_docs/version-3.7/experiments/request_locking.mdx
index 30c9cd74d..f4f00116f 100644
--- a/website/versioned_docs/version-3.7/experiments/request_locking.mdx
+++ b/website/versioned_docs/version-3.7/experiments/request_locking.mdx
@@ -21,7 +21,7 @@ API. With this API, you will be able to pass a `RequestQueue` to multiple crawle
:::info Keep in mind
-The request queue that supports request locking is currently exported via the `RequestQueue2` class. Once the experiment is over, this class will replace
+The request queue that supports request locking is currently exported via the `RequestQueueV2` class. Once the experiment is over, this class will replace
the current `RequestQueue` class
:::
@@ -58,13 +58,13 @@ await crawler.run(['https://crawlee.dev']);
### Outside crawlers (to setup your own request queue that supports locking)
-Previously, you would import `RequestQueue` from `crawlee`. To switch to the queue that supports locking, you need to import `RequestQueue2` instead.
+Previously, you would import `RequestQueue` from `crawlee`. To switch to the queue that supports locking, you need to import `RequestQueueV2` instead.
```ts
// highlight-next-line
-import { RequestQueue2 } from 'crawlee';
+import { RequestQueueV2 } from 'crawlee';
-const queue = await RequestQueue2.open('my-locking-queue');
+const queue = await RequestQueueV2.open('my-locking-queue');
await queue.addRequests([
{ url: 'https://crawlee.dev' },
{ url: 'https://crawlee.dev/docs' },
@@ -83,10 +83,10 @@ If you do not enable the experiment, you will receive a runtime error and the cr
:::
```ts
-import { CheerioCrawler, RequestQueue2 } from 'crawlee';
+import { CheerioCrawler, RequestQueueV2 } from 'crawlee';
// highlight-next-line
-const queue = await RequestQueue2.open('my-locking-queue');
+const queue = await RequestQueueV2.open('my-locking-queue');
const crawler = new CheerioCrawler({
// highlight-next-line
diff --git a/website/versioned_docs/version-3.8/experiments/request_locking.mdx b/website/versioned_docs/version-3.8/experiments/request_locking.mdx
index 30c9cd74d..f4f00116f 100644
--- a/website/versioned_docs/version-3.8/experiments/request_locking.mdx
+++ b/website/versioned_docs/version-3.8/experiments/request_locking.mdx
@@ -21,7 +21,7 @@ API. With this API, you will be able to pass a `RequestQueue` to multiple crawle
:::info Keep in mind
-The request queue that supports request locking is currently exported via the `RequestQueue2` class. Once the experiment is over, this class will replace
+The request queue that supports request locking is currently exported via the `RequestQueueV2` class. Once the experiment is over, this class will replace
the current `RequestQueue` class
:::
@@ -58,13 +58,13 @@ await crawler.run(['https://crawlee.dev']);
### Outside crawlers (to setup your own request queue that supports locking)
-Previously, you would import `RequestQueue` from `crawlee`. To switch to the queue that supports locking, you need to import `RequestQueue2` instead.
+Previously, you would import `RequestQueue` from `crawlee`. To switch to the queue that supports locking, you need to import `RequestQueueV2` instead.
```ts
// highlight-next-line
-import { RequestQueue2 } from 'crawlee';
+import { RequestQueueV2 } from 'crawlee';
-const queue = await RequestQueue2.open('my-locking-queue');
+const queue = await RequestQueueV2.open('my-locking-queue');
await queue.addRequests([
{ url: 'https://crawlee.dev' },
{ url: 'https://crawlee.dev/docs' },
@@ -83,10 +83,10 @@ If you do not enable the experiment, you will receive a runtime error and the cr
:::
```ts
-import { CheerioCrawler, RequestQueue2 } from 'crawlee';
+import { CheerioCrawler, RequestQueueV2 } from 'crawlee';
// highlight-next-line
-const queue = await RequestQueue2.open('my-locking-queue');
+const queue = await RequestQueueV2.open('my-locking-queue');
const crawler = new CheerioCrawler({
// highlight-next-line