Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

fix(RequestQueueV2): reset recently handled cache too if the queue is pending for too long (#2656)

Vlad Frangu51a69bc1f208
diff --git a/packages/core/src/storages/request_provider.ts b/packages/core/src/storages/request_provider.ts
index 45f1466d4..75280010a 100644
--- a/packages/core/src/storages/request_provider.ts
+++ b/packages/core/src/storages/request_provider.ts
@@ -571,6 +571,8 @@ export abstract class RequestProvider implements IStorage {
});
this.queueHeadIds.clear();
+ // This cache may be the bane of our existence, but it's still required for v1...
+ this.recentlyHandledRequestsCache.clear();
}
if (this.queueHeadIds.length() > 0) {