Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

RequestLikeBrowser does not throw on empty body now.

Petr Patek93bcd95bd037
diff --git a/src/utils_request.js b/src/utils_request.js
index ee16e1fbf..155747139 100644
--- a/src/utils_request.js
+++ b/src/utils_request.js
@@ -316,14 +316,7 @@ export const requestLikeBrowser = async (options) => {
opts.headers = _.defaults({}, opts.headers, browserHeaders);
opts.abortFunction = abortFunction;
- const response = await requestExtended(opts);
-
-
- const { body } = response;
-
- if (!body) throw new Error('The response body is empty');
-
- return response;
+ return requestExtended(opts);
};
/**
diff --git a/test/utils_request.js b/test/utils_request.js
index 28072d788..d9b775d86 100644
--- a/test/utils_request.js
+++ b/test/utils_request.js
@@ -379,7 +379,7 @@ describe('Apify.utils_request', () => {
expect(error.message).to.eql(`utils.requestBetter: Request for ${options.url} aborted due to abortFunction`);
});
- it('it throws for empty response body', async () => {
+ it('it does not throw for empty response body', async () => {
const options = {
url: `http://${HOST}:${port}/empty`,
};
@@ -390,8 +390,7 @@ describe('Apify.utils_request', () => {
error = e;
}
- expect(error).to.exist; //eslint-disable-line
- expect(error.message).to.eql('The response body is empty');
+ expect(error).to.not.exist; //eslint-disable-line
});
it('it throws for other contentType then - text/html', async () => {