Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

fix: do not use `tryCancel()` from inside sync callback (#1265)

Martin Adámek6f951f14a47f
diff --git a/src/storages/request_queue.js b/src/storages/request_queue.js
index 12a9189b7..462881b2b 100644
--- a/src/storages/request_queue.js
+++ b/src/storages/request_queue.js
@@ -13,7 +13,7 @@ import Request, { RequestOptions } from '../request'; // eslint-disable-line imp
import { ApifyClient } from 'apify-client';
// @ts-ignore
import { ApifyStorageLocal } from '@apify/storage-local';
-import { tryCancel } from '@apify/timeout';
+import { storage } from '@apify/timeout';
/* eslint-enable no-unused-vars,import/named,import/no-duplicates,import/order */
const MAX_CACHED_REQUESTS = 1000 * 1000;
@@ -410,7 +410,11 @@ export class RequestQueue {
// Wait a little to increase a chance that the next call to fetchNextRequest() will return the request with updated data.
// This is to compensate for the limitation of DynamoDB, where writes might not be immediately visible to subsequent reads.
setTimeout(() => {
- tryCancel();
+ const signal = storage.getStore()?.cancelTask?.signal;
+
+ if (signal?.aborted) {
+ return;
+ }
if (!this.inProgress.has(request.id)) {
this.log.debug('The request is no longer marked as in progress in the queue?!', { requestId: request.id });