Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

feat(http): add an ignoreTlsErrors option to FileDownload (#4362)

Vlad Frangu1fa8d39dcd8b
diff --git a/docs/public-api/crawlee-http.api.md b/docs/public-api/crawlee-http.api.md
index d886700e5..dc1f31444 100644
--- a/docs/public-api/crawlee-http.api.md
+++ b/docs/public-api/crawlee-http.api.md
@@ -102,7 +102,7 @@ export interface DOMParseResult {
// @public
export class FileDownload extends BasicCrawler<FileDownloadCrawlingContext> {
- constructor(options?: Omit<BasicCrawlerOptions<FileDownloadCrawlingContext>, 'contextPipelineBuilder'>);
+ constructor(options?: FileDownloadOptions);
}
// @public (undocumented)
@@ -122,6 +122,11 @@ export interface FileDownloadCrawlingContext<UserData extends Dictionary = any>
export type FileDownloadErrorHandler<UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
ContextExtension = Dictionary<never>> = ErrorHandler<CrawlingContext, FileDownloadCrawlingContext<UserData> & ContextExtension>;
+// @public (undocumented)
+export interface FileDownloadOptions extends Omit<BasicCrawlerOptions<FileDownloadCrawlingContext>, 'contextPipelineBuilder'> {
+ ignoreTlsErrors?: boolean;
+}
+
// @public (undocumented)
export type FileDownloadRequestHandler<UserData extends Dictionary = any> = RequestHandler<FileDownloadCrawlingContext<UserData>>;
diff --git a/docs/upgrading/upgrading-v4/removed-symbols.md b/docs/upgrading/upgrading-v4/removed-symbols.md
index ce0f1496f..72f5904cb 100644
--- a/docs/upgrading/upgrading-v4/removed-symbols.md
+++ b/docs/upgrading/upgrading-v4/removed-symbols.md
@@ -37,7 +37,7 @@ The full list of removed exports and members, for ctrl-F purposes. Where a repla
- `checkAndSerialize` and `chunkBySize` functions (from `@crawlee/core`) — value (de)serialization now lives in the `KeyValueStore` frontend; use `serializeValue` / `parseValue` (see [`maybeStringify` is removed](./storage-backends.md#maybestringify-is-removed))
- `BASIC_CRAWLER_TIMEOUT_BUFFER_SECS` constant (from `@crawlee/basic`) — was an internal timeout buffer, no longer exported
- `HttpResponse`, `HttpResponseWithoutBody`, `StreamingHttpResponse`, `ResponseTypes`, `BaseHttpResponseData`, `SimpleHeaders`, `processHttpRequestOptions`, and `GotScrapingHttpClient` (from `@crawlee/core`) — the HTTP client surface moved to `@crawlee/http-client` / `@crawlee/got-scraping-client` (see [HTTP client packages and `BaseHttpClient` reshaped](./http-clients.md#http-client-packages-and-basehttpclient-reshaped))
-- `StreamHandlerContext` and `FileDownloadOptions` types (from `@crawlee/http`) — see [`FileDownload` now extends `BasicCrawler`](./upgrading-v4.md#filedownload-now-extends-basiccrawler-and-no-longer-takes-filedownloadoptions)
+- `StreamHandlerContext` type (from `@crawlee/http`); `FileDownloadOptions` stays but only adds `ignoreTlsErrors` on top of `BasicCrawlerOptions` — see [`FileDownload` now extends `BasicCrawler`](./upgrading-v4.md#filedownload-now-extends-basiccrawler-and-takes-a-slimmer-filedownloadoptions)
- `PlainResponse` type (from `@crawlee/http`) — it wrapped the `got-scraping` response and is gone along with the rest of the old HTTP response surface (see [`CrawlingContext.response` is now of type `Response`](./upgrading-v4.md#crawlingcontextresponse-is-now-of-type-response))
- `checkStorageAccess`, `withCheckedStorageAccess` and the `RequestHandlerResult` type — superseded by the storage transaction mechanism; use `withDirectStorageAccess()` and `StorageTransactionView` (see [Storage writes in request handlers are transactional](./upgrading-v4.md#storage-writes-in-request-handlers-are-transactional))
- `CreateContextOptions` type (from `@crawlee/basic`) — a leftover of the pre-`ContextPipeline` context-creation design, unused by the library itself; context construction is now driven by `ContextPipeline`
diff --git a/docs/upgrading/upgrading-v4/upgrading-v4.md b/docs/upgrading/upgrading-v4/upgrading-v4.md
index 4a93e8cec..61820417c 100644
--- a/docs/upgrading/upgrading-v4/upgrading-v4.md
+++ b/docs/upgrading/upgrading-v4/upgrading-v4.md
@@ -613,7 +613,7 @@ Building the pool outside the crawler has one consequence worth knowing: a pool
The crawler option is renamed to `ignoreTlsErrors`, matching the naming used everywhere else in v4 (`session.proxyInfo.ignoreTlsErrors`, the browser pool, the impit client). The old `ignoreSslErrors` name is no longer accepted — rename it in your crawler options.
-The default also changed from `true` to `false`: HTTP crawlers now verify TLS certificates and fail requests to sites with invalid ones. Set `ignoreTlsErrors: true` to restore the v3 behavior.
+The default also changed from `true` to `false`: HTTP crawlers now verify TLS certificates and fail requests to sites with invalid ones. Set `ignoreTlsErrors: true` to restore the v3 behavior. `FileDownload` takes the same option with the same default; see [its section](#filedownload-now-extends-basiccrawler-and-takes-a-slimmer-filedownloadoptions).
Under the hood the crawler now forwards the option to the HTTP client as `SendRequestOptions.ignoreTlsErrors` on every navigation request, and the same flag is enabled automatically for MITM proxy sessions (`session.proxyInfo.ignoreTlsErrors`), matching the browser crawlers.
@@ -1158,9 +1158,9 @@ See [Upgrading to v4: HTTP clients](./http-clients.md) for the details.
Applies when you use the `FileDownload` crawler from `@crawlee/http`.
-### `FileDownload` now extends `BasicCrawler` and no longer takes `FileDownloadOptions`
+### `FileDownload` now extends `BasicCrawler` and takes a slimmer `FileDownloadOptions`
-`FileDownload` was re-based from `HttpCrawler` onto `BasicCrawler`. Its constructor now accepts `BasicCrawlerOptions<FileDownloadCrawlingContext>` instead of the dedicated `FileDownloadOptions` type, which — together with `StreamHandlerContext` — has been **removed** from `@crawlee/http`. In practice this means the HTTP-crawler-specific options (`navigationTimeoutSecs`, `additionalMimeTypes`, `suggestResponseEncoding`, `forceResponseEncoding`, the `gotOptions`-style `preNavigationHooks`, etc.) are no longer accepted by `FileDownload`; downloading is a thin layer over `BasicCrawler` and the request is performed via `sendRequest` / the configured `httpClient`. If you passed any of those HTTP-only options to `FileDownload`, drop them and configure the `httpClient` (or the request itself) directly. The `FileDownloadCrawlingContext` type also lost its extra type parameter and no longer extends the internal HTTP crawling context — it now extends the common `CrawlingContext` with `contentType`, `request`, and `response`.
+`FileDownload` was re-based from `HttpCrawler` onto `BasicCrawler`. `FileDownloadOptions` is now `BasicCrawlerOptions<FileDownloadCrawlingContext>` (minus `contextPipelineBuilder`) plus a single `ignoreTlsErrors` flag; `StreamHandlerContext` has been **removed** from `@crawlee/http`. In practice this means the HTTP-crawler-specific options (`navigationTimeoutSecs`, `additionalMimeTypes`, `suggestResponseEncoding`, `forceResponseEncoding`, the `gotOptions`-style `preNavigationHooks`, etc.) are no longer accepted by `FileDownload`; downloading is a thin layer over `BasicCrawler` and the request is performed via `sendRequest` / the configured `httpClient`. If you passed any of those HTTP-only options to `FileDownload`, drop them and configure the `httpClient` (or the request itself) directly. TLS verification changed too: v3 `FileDownload` inherited `ignoreSslErrors: true`, while v4 verifies certificates by default (`ignoreTlsErrors` defaults to `false`). Sessions with a MITM proxy (`session.proxyInfo.ignoreTlsErrors`) still skip verification automatically. To keep the v3 behavior, pass `ignoreTlsErrors: true`. The `FileDownloadCrawlingContext` type also lost its extra type parameter and no longer extends the internal HTTP crawling context — it now extends the common `CrawlingContext` with `contentType`, `request`, and `response`.
### Crawling context in the `FileDownload` crawler no longer includes `body` and `stream` properties
diff --git a/packages/http-crawler/src/internals/file-download.ts b/packages/http-crawler/src/internals/file-download.ts
index 8b9f07729..f7f2b9c60 100644
--- a/packages/http-crawler/src/internals/file-download.ts
+++ b/packages/http-crawler/src/internals/file-download.ts
@@ -8,6 +8,8 @@ import type {
import { BasicCrawler, ContextPipeline } from '@crawlee/basic';
import { ResponseWithUrl } from '@crawlee/http-client';
import type { Dictionary } from '@crawlee/types';
+import { parseArgument } from '@crawlee/utils/internal';
+import { z } from 'zod';
import type {
ErrorHandler,
@@ -38,6 +40,24 @@ export type FileDownloadRequestHandler<
UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
> = RequestHandler<FileDownloadCrawlingContext<UserData>>;
+export interface FileDownloadOptions extends Omit<
+ BasicCrawlerOptions<FileDownloadCrawlingContext>,
+ 'contextPipelineBuilder'
+> {
+ /**
+ * If set to `true`, TLS/SSL certificate errors are ignored. Forwarded to the HTTP client as
+ * {@apilink SendRequestOptions.ignoreTlsErrors|`ignoreTlsErrors`} on every download, so custom
+ * {@apilink BaseHttpClient} implementations should honor that flag (the built-in impit and got-scraping
+ * clients do; the native fetch fallback cannot disable TLS verification and warns instead).
+ *
+ * Certificates are verified by default. Sessions whose proxy is a MITM proxy
+ * (`session.proxyInfo.ignoreTlsErrors`) skip verification regardless of this option.
+ *
+ * @default false
+ */
+ ignoreTlsErrors?: boolean;
+}
+
/**
* Provides a framework for downloading files in parallel using plain HTTP requests. The URLs to download are fed either from a static list of URLs or they can be added on the fly from another crawler.
*
@@ -72,12 +92,34 @@ export type FileDownloadRequestHandler<
* ```
*/
export class FileDownload extends BasicCrawler<FileDownloadCrawlingContext> {
+ #ignoreTlsErrors: boolean;
+
+ /**
+ * @internal
+ */
+ protected static override optionsShape = {
+ ...BasicCrawler.optionsShape,
+
+ ignoreTlsErrors: z.boolean().default(false),
+ };
+
+ /** @internal */
+ protected static optionsSchema = z.strictObject(FileDownload.optionsShape);
+
// TODO hooks
- constructor(options: Omit<BasicCrawlerOptions<FileDownloadCrawlingContext>, 'contextPipelineBuilder'> = {}) {
+ constructor(options: FileDownloadOptions = {}) {
+ const { ignoreTlsErrors, ...basicCrawlerOptions } = parseArgument(
+ options,
+ FileDownload.optionsSchema,
+ 'FileDownloadOptions',
+ );
+
super({
- ...options,
+ ...basicCrawlerOptions,
contextPipelineBuilder: () => this.#buildContextPipeline(),
});
+
+ this.#ignoreTlsErrors = ignoreTlsErrors;
}
#buildContextPipeline(): ContextPipeline<CrawlingContext, FileDownloadCrawlingContext> {
@@ -87,6 +129,7 @@ export class FileDownload extends BasicCrawler<FileDownloadCrawlingContext> {
async #initiateDownload(context: CrawlingContext, onCleanup: CleanupRegistrar) {
const response = await this.httpClient.sendRequest(context.request.intoFetchAPIRequest(), {
session: context.session,
+ ignoreTlsErrors: this.#ignoreTlsErrors,
});
const { type, charset: encoding } = parseContentTypeFromResponse(response);
diff --git a/test/core/crawlers/file-download.test.ts b/test/core/crawlers/file-download.test.ts
index e3806065f..bfbda4bb5 100644
--- a/test/core/crawlers/file-download.test.ts
+++ b/test/core/crawlers/file-download.test.ts
@@ -200,3 +200,31 @@ test('crawler waits for the stream to be consumed', async () => {
expect(bufferedData.length).toBe(5 * 1024);
expect(bufferedData).toEqual(await ReadableStreamGenerator.getUint8Array(5 * 1024, 789));
});
+
+test('verifies TLS certificates by default', async () => {
+ const crawler = new FileDownload({
+ maxRequestRetries: 0,
+ requestHandler: async () => {},
+ });
+
+ const stats = await crawler.run(['https://self-signed.badssl.com/']);
+
+ expect(stats.requestsFailed).toBe(1);
+});
+
+test('ignoreTlsErrors: true accepts invalid certificates', async () => {
+ let status: number | undefined;
+
+ const crawler = new FileDownload({
+ maxRequestRetries: 0,
+ ignoreTlsErrors: true,
+ requestHandler: async ({ response }) => {
+ status = response.status;
+ },
+ });
+
+ const stats = await crawler.run(['https://self-signed.badssl.com/']);
+
+ expect(stats.requestsFailed).toBe(0);
+ expect(status).toBe(200);
+});