Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
fix(http)!: verify TLS certificates by default (#4317)
Vlad Frangu5a889d2733f0
diff --git a/docs/upgrading/upgrading-v4/upgrading-v4.md b/docs/upgrading/upgrading-v4/upgrading-v4.mdindex 7793c2e15..576c846ae 100644--- a/docs/upgrading/upgrading-v4/upgrading-v4.md+++ b/docs/upgrading/upgrading-v4/upgrading-v4.md@@ -609,9 +609,11 @@ Building the pool outside the crawler has one consequence worth knowing: a pool`headless` is now declared on each concrete crawler rather than on `BrowserCrawler`, so the puppeteer-only `'new'` and `'old'` values are only accepted by `PuppeteerCrawler`.-### `ignoreSslErrors` is renamed to `ignoreTlsErrors`+### `ignoreSslErrors` is renamed to `ignoreTlsErrors` and defaults to `false`-The crawler option is renamed to `ignoreTlsErrors`, matching the naming used everywhere else in v4 (`session.proxyInfo.ignoreTlsErrors`, the browser pool, the impit client). The old `ignoreSslErrors` name is no longer accepted — rename it in your crawler options. Behavior is unchanged from v3: the option defaults to `true` and HTTP crawlers accept invalid TLS certificates by default.+The crawler option is renamed to `ignoreTlsErrors`, matching the naming used everywhere else in v4 (`session.proxyInfo.ignoreTlsErrors`, the browser pool, the impit client). The old `ignoreSslErrors` name is no longer accepted — rename it in your crawler options.++The default also changed from `true` to `false`: HTTP crawlers now verify TLS certificates and fail requests to sites with invalid ones. Set `ignoreTlsErrors: true` to restore the v3 behavior.Under the hood the crawler now forwards the option to the HTTP client as `SendRequestOptions.ignoreTlsErrors` on every navigation request, and the same flag is enabled automatically for MITM proxy sessions (`session.proxyInfo.ignoreTlsErrors`), matching the browser crawlers.diff --git a/packages/http-crawler/src/internals/http-crawler.ts b/packages/http-crawler/src/internals/http-crawler.tsindex 500b4c5a0..3ecdf2e3c 100644--- a/packages/http-crawler/src/internals/http-crawler.ts+++ b/packages/http-crawler/src/internals/http-crawler.ts@@ -99,7 +99,7 @@ export interface HttpCrawlerOptions<* {@apilink BaseHttpClient} implementations should honor that flag (the built-in impit and got-scraping* clients do; the native fetch fallback cannot disable TLS verification and warns instead).*- * @default true+ * @default false*/ignoreTlsErrors?: boolean;@@ -386,7 +386,7 @@ export class HttpCrawler<...BasicCrawler.optionsShape,navigationTimeoutSecs: schemas.anyNumber.default(30),- ignoreTlsErrors: z.boolean().default(true),+ ignoreTlsErrors: z.boolean().default(false),additionalMimeTypes: schemas.arrayOf(z.string(), 'strings').default(() => []),suggestResponseEncoding: z.string().optional(),forceResponseEncoding: z.string().optional(),diff --git a/test/core/crawlers/cheerio-crawler.test.ts b/test/core/crawlers/cheerio-crawler.test.tsindex 7a4a73ef9..56246ca1a 100644--- a/test/core/crawlers/cheerio-crawler.test.ts+++ b/test/core/crawlers/cheerio-crawler.test.ts@@ -224,7 +224,15 @@ describe('CheerioCrawler', () => {});await strictCrawler.run([`${serverAddress}/?tls=strict`]);- expect(captured).toEqual([true, false]);+ const lenientCrawler = new CheerioCrawler({+ httpClient: capturingClient,+ ignoreTlsErrors: true,+ maxConcurrency: 1,+ requestHandler: () => {},+ });+ await lenientCrawler.run([`${serverAddress}/?tls=lenient`]);++ expect(captured).toEqual([false, false, true]);});test('should work with skipNavigation', async () => {