Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Fixed RequestList code samples, better log messages

Jan Curn3da1fb32aa3c
diff --git a/src/basic_crawler.js b/src/basic_crawler.js
index e3bc74ddb..b422e1c84 100644
--- a/src/basic_crawler.js
+++ b/src/basic_crawler.js
@@ -189,7 +189,7 @@ export default class BasicCrawler {
// Retry request.
if (request.retryCount < this.maxRequestRetries) {
request.retryCount++;
- log.exception(error, 'RequestQueue: handleRequestFunction failed, reclaiming failed request back to queue', {
+ log.exception(error, 'BasicCrawler: handleRequestFunction failed, reclaiming failed request back to list/queue', {
url: request.url,
retryCount: request.retryCount,
});
@@ -197,7 +197,7 @@ export default class BasicCrawler {
return source.reclaimRequest(request);
}
- log.exception(error, 'RequestQueue: handleRequestFunction failed, marking failed request handled', {
+ log.exception(error, 'BasicCrawler: handleRequestFunction failed, marking failed request handled', {
url: request.url,
retryCount: request.retryCount,
});
diff --git a/src/request_list.js b/src/request_list.js
index f6398d7d1..b2214b31d 100644
--- a/src/request_list.js
+++ b/src/request_list.js
@@ -53,15 +53,15 @@ const ensureUniqueKeyValid = (uniqueKey) => {
* await requestList.initialize(); // Load requests.
*
* // Get requests from list
- * const request1 = requestList.fetchNextRequest();
- * const request2 = requestList.fetchNextRequest();
- * const request3 = requestList.fetchNextRequest();
+ * const request1 = await requestList.fetchNextRequest();
+ * const request2 = await requestList.fetchNextRequest();
+ * const request3 = await requestList.fetchNextRequest();
*
* // Mark some of them as handled
- * requestList.markRequestHandled(request1);
+ * await requestList.markRequestHandled(request1);
*
* // If processing fails then reclaim it back to the list
- * requestList.reclaimRequest(request2);
+ * await requestList.reclaimRequest(request2);
* ```
*
* @param {Object} options