Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

fix: retry on blocked status codes in `HttpCrawler` (#3060)

Jindřich Bärb5fcd79324ed
diff --git a/packages/http-crawler/src/internals/http-crawler.ts b/packages/http-crawler/src/internals/http-crawler.ts
index 1811a6b35..f7ce338f8 100644
--- a/packages/http-crawler/src/internals/http-crawler.ts
+++ b/packages/http-crawler/src/internals/http-crawler.ts
@@ -19,6 +19,7 @@ import type {
import {
BASIC_CRAWLER_TIMEOUT_BUFFER_SECS,
BasicCrawler,
+ BLOCKED_STATUS_CODES,
Configuration,
CrawlerExtension,
mergeCookies,
@@ -602,6 +603,18 @@ export class HttpCrawler<
return `Found selectors: ${foundSelectors.join(', ')}`;
}
}
+
+ const blockedStatusCodes =
+ // eslint-disable-next-line dot-notation
+ (this.sessionPool?.['blockedStatusCodes'].length ?? 0) > 0
+ ? // eslint-disable-next-line dot-notation
+ this.sessionPool!['blockedStatusCodes']
+ : BLOCKED_STATUS_CODES;
+
+ if (blockedStatusCodes.includes(crawlingContext.response.statusCode!)) {
+ return `Blocked by status code ${crawlingContext.response.statusCode}`;
+ }
+
return false;
}