Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Cheerio ignores ssl errors by default now.

Petr Patek2cf99c18b228
diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.md
index 0fe49ae90..6aa8b9399 100644
--- a/docs/api/CheerioCrawler.md
+++ b/docs/api/CheerioCrawler.md
@@ -184,7 +184,7 @@ The exceptions are logged to the request using the
<tr>
<td colspan="3"><p>Timeout in which the HTTP request to the resource needs to finish, given in seconds.</p>
</td></tr><tr>
-<td><code>[options.ignoreSslErrors]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
+<td><code>[options.ignoreSslErrors]</code></td><td><code>Boolean</code></td><td><code>true</code></td>
</tr>
<tr>
<td colspan="3"><p>If set to true, SSL certificate errors will be ignored.</p>
diff --git a/src/crawlers/cheerio_crawler.js b/src/crawlers/cheerio_crawler.js
index 9e879b63c..0e9f85a8f 100644
--- a/src/crawlers/cheerio_crawler.js
+++ b/src/crawlers/cheerio_crawler.js
@@ -28,7 +28,7 @@ const DEFAULT_OPTIONS = {
log.error('CheerioCrawler: Request failed and reached maximum retries', details);
},
- ignoreSslErrors: false,
+ ignoreSslErrors: true,
useApifyProxy: false,
autoscaledPoolOptions: {
snapshotterOptions: {
@@ -200,7 +200,7 @@ const DEFAULT_OPTIONS = {
* Timeout in which the function passed as `options.handlePageFunction` needs to finish, given in seconds.
* @param {Number} [options.requestTimeoutSecs=30]
* Timeout in which the HTTP request to the resource needs to finish, given in seconds.
- * @param {Boolean} [options.ignoreSslErrors=false]
+ * @param {Boolean} [options.ignoreSslErrors=true]
* If set to true, SSL certificate errors will be ignored.
* @param {Boolean} [options.useApifyProxy=false]
* If set to `true`, `CheerioCrawler` will be configured to use
diff --git a/test/crawlers/cheerio_crawler.test.js b/test/crawlers/cheerio_crawler.test.js
index 588b041eb..d55783f55 100644
--- a/test/crawlers/cheerio_crawler.test.js
+++ b/test/crawlers/cheerio_crawler.test.js
@@ -190,6 +190,25 @@ describe('CheerioCrawler', () => {
});
});
+ test('should ignore ssl by default', async () => {
+ const sources = [
+ { url: 'http://example.com/?q=1' },
+ ];
+ const requestList = new Apify.RequestList({ sources });
+ const handlePageFunction = async () => {};
+
+ const cheerioCrawler = new Apify.CheerioCrawler({
+ requestList,
+ maxConcurrency: 1,
+ handlePageFunction,
+ });
+
+ await requestList.initialize();
+ await cheerioCrawler.run();
+
+ expect(cheerioCrawler.ignoreSslErrors).toBeTruthy();
+ });
+
test('should trigger prepareRequestFunction', async () => {
const MODIFIED_URL = 'http://example.com/?q=2';
const sources = [