Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Added a TODO

Jan Curnc23d13e3473e
diff --git a/src/request_queue.js b/src/request_queue.js
index 84d0409b3..eff12bf37 100644
--- a/src/request_queue.js
+++ b/src/request_queue.js
@@ -305,6 +305,7 @@ export class RequestQueue {
return this
.getRequest(nextId)
.then((request) => {
+ // TODO: What happens when getRequest() fails? Shouldn't we run the code below too???
// We need to handle this situation because request may not be available
// immediately after adding to the queue.
if (!request) {