Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

fix(utils): treat a missing robots.txt (HTTP 404) as allow-all (#4171)

brekenf3fd76482f2e
diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.md
index 6c338619e..95fc60d19 100644
--- a/docs/upgrading/upgrading_v4.md
+++ b/docs/upgrading/upgrading_v4.md
@@ -904,6 +904,12 @@ The `robotsTxtFile` / `respectRobotsTxtFile` per-call options are removed from `
The callback now gets `{ request, reason }` instead of `{ url, reason }` — use `request.url` for the URL.
+### robots.txt error responses follow RFC 9309
+
+v3 parsed the body of a robots.txt response regardless of its status code. v4 follows [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309#section-2.3.1.3) instead: a `4xx` response allows everything and a `5xx` response disallows everything, whatever the body says.
+
+With `respectRobotsTxtFile` enabled, a site that answers `/robots.txt` with a `5xx` is therefore skipped entirely. The crawler caches robots.txt per origin for the whole run, so every request to that origin is skipped with the `robotsTxt` reason.
+
### Internal KVS keys renamed
Several internal Crawlee keys were prefixed with the `SDK_` prefix for legacy reasons — these keys now start with `CRAWLEE_` instead. These are, e.g., `CRAWLEE_SESSION_POOL_STATE` or `CRAWLEE_CRAWLER_STATISTICS_{n}`.
diff --git a/packages/utils/src/internals/robots.ts b/packages/utils/src/internals/robots.ts
index 397f8e9d2..81f41ee30 100644
--- a/packages/utils/src/internals/robots.ts
+++ b/packages/utils/src/internals/robots.ts
@@ -106,16 +106,14 @@ export class RobotsTxtFile {
signal: options?.signal,
});
- if (response.status < 200 || response.status >= 300) {
- throw new Error(`Failed to load robots.txt from ${url}: HTTP ${response.status}`);
- }
-
- if (response.status === 404) {
+ // RFC 9309: an unavailable (4xx) robots.txt allows everything, an unreachable (5xx) one disallows everything.
+ if (response.status >= 400 && response.status < 600) {
+ const allowed = response.status < 500;
return new RobotsTxtFile(
url,
{
isAllowed() {
- return true;
+ return allowed;
},
getSitemaps() {
return [];
@@ -129,6 +127,10 @@ export class RobotsTxtFile {
);
}
+ if (response.status < 200 || response.status >= 300) {
+ throw new Error(`Failed to load robots.txt from ${url}: HTTP ${response.status}`);
+ }
+
// @ts-ignore
return new RobotsTxtFile(url, robotsParser(url.toString(), await response.text()), proxyUrl, logger);
}
diff --git a/packages/utils/test/robots.test.ts b/packages/utils/test/robots.test.ts
index 789be7f33..e1cd969c0 100644
--- a/packages/utils/test/robots.test.ts
+++ b/packages/utils/test/robots.test.ts
@@ -67,6 +67,32 @@ describe('RobotsTxtFile', () => {
]);
});
+ it.each([401, 403, 404, 410])('treats an unavailable robots.txt (HTTP %i) as allow-all', async (status) => {
+ nock('http://no-robots.com').get('/robots.txt').reply(status, 'User-agent: *\nDisallow: /');
+
+ const robots = await RobotsTxtFile.find('http://no-robots.com/some/page.html', { httpClient });
+
+ expect(robots.isAllowed('http://no-robots.com/some/page.html')).toBe(true);
+ expect(robots.getSitemaps()).toEqual([]);
+ expect(robots.getCrawlDelay()).toBeUndefined();
+ });
+
+ it.each([500, 503])('treats an unreachable robots.txt (HTTP %i) as disallow-all', async (status) => {
+ nock('http://broken-robots.com').get('/robots.txt').reply(status, 'User-agent: *\nAllow: /');
+
+ const robots = await RobotsTxtFile.find('http://broken-robots.com/some/page.html', { httpClient });
+
+ expect(robots.isAllowed('http://broken-robots.com/some/page.html')).toBe(false);
+ expect(robots.getSitemaps()).toEqual([]);
+ expect(robots.getCrawlDelay()).toBeUndefined();
+ });
+
+ it('still rejects on other non-2xx responses', async () => {
+ nock('http://broken-robots.com').get('/robots.txt').reply(304);
+
+ await expect(RobotsTxtFile.find('http://broken-robots.com/', { httpClient })).rejects.toThrow(/HTTP 304/);
+ });
+
it('respects user-set timeout', async () => {
const start = Date.now();
const robots = RobotsTxtFile.find(`${neverReplies}/robots.txt`, { timeoutMillis: 200 });