Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

chore: wire the `ignoreTlsErrors` crawler option (formerly `ignoreSslErrors`) through to the HTTP client (#4011)

Martin Adámekd48cd5b85578
diff --git a/docs/public-api/crawlee-http-client.api.md b/docs/public-api/crawlee-http-client.api.md
index d0f4888c0..ce9b5e432 100644
--- a/docs/public-api/crawlee-http-client.api.md
+++ b/docs/public-api/crawlee-http-client.api.md
@@ -23,11 +23,15 @@ export abstract class BaseHttpClient implements BaseHttpClient_2 {
export interface CustomFetchOptions {
cookieJar?: CookieJar;
fingerprint?: SessionFingerprint;
+ ignoreTlsErrors?: boolean;
proxyUrl?: string;
}
// @public
export class FetchHttpClient extends BaseHttpClient {
+ constructor(options?: {
+ logger?: CrawleeLogger;
+ });
// (undocumented)
fetch(request: Request, options?: RequestInit & CustomFetchOptions): Promise<Response>;
}
diff --git a/docs/public-api/crawlee-http.api.md b/docs/public-api/crawlee-http.api.md
index 4fe3c6d45..4eff6be95 100644
--- a/docs/public-api/crawlee-http.api.md
+++ b/docs/public-api/crawlee-http.api.md
@@ -117,7 +117,7 @@ export class HttpCrawler<Context extends InternalHttpCrawlingContext<any, any> =
// (undocumented)
protected static optionsShape: {
navigationTimeoutSecs: NumberPredicate & BasePredicate<number | undefined>;
- ignoreSslErrors: BooleanPredicate & BasePredicate<boolean | undefined>;
+ ignoreTlsErrors: BooleanPredicate & BasePredicate<boolean | undefined>;
additionalMimeTypes: ArrayPredicate<string>;
suggestResponseEncoding: StringPredicate & BasePredicate<string | undefined>;
forceResponseEncoding: StringPredicate & BasePredicate<string | undefined>;
@@ -168,7 +168,7 @@ export class HttpCrawler<Context extends InternalHttpCrawlingContext<any, any> =
export interface HttpCrawlerOptions<Context extends InternalHttpCrawlingContext = InternalHttpCrawlingContext, ContextExtension = Dictionary<never>, ExtendedContext extends Context = Context & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>> extends BasicCrawlerOptions<Context, ContextExtension, ExtendedContext, Routes> {
additionalMimeTypes?: string[];
forceResponseEncoding?: string;
- ignoreSslErrors?: boolean;
+ ignoreTlsErrors?: boolean;
navigationTimeoutSecs?: number;
postNavigationHooks?: ((crawlingContext: CrawlingContextWithResponse & ContextExtension) => Awaitable<void | Partial<CrawlingContextWithResponse>>)[];
preNavigationHooks?: InternalHttpHook<CrawlingContext<any>, ContextExtension>[];
diff --git a/docs/public-api/crawlee-jsdom.api.md b/docs/public-api/crawlee-jsdom.api.md
index 2338e4fb9..1a22baf1d 100644
--- a/docs/public-api/crawlee-jsdom.api.md
+++ b/docs/public-api/crawlee-jsdom.api.md
@@ -61,7 +61,7 @@ export class JSDOMCrawler<ContextExtension = Dictionary<never>, ExtendedContext
runScripts: BooleanPredicate & BasePredicate<boolean | undefined>;
hideInternalConsole: BooleanPredicate & BasePredicate<boolean | undefined>;
navigationTimeoutSecs: NumberPredicate & BasePredicate<number | undefined>;
- ignoreSslErrors: BooleanPredicate & BasePredicate<boolean | undefined>;
+ ignoreTlsErrors: BooleanPredicate & BasePredicate<boolean | undefined>;
additionalMimeTypes: ArrayPredicate<string>;
suggestResponseEncoding: StringPredicate & BasePredicate<string | undefined>;
forceResponseEncoding: StringPredicate & BasePredicate<string | undefined>;
diff --git a/docs/public-api/crawlee-types.api.md b/docs/public-api/crawlee-types.api.md
index 87cac4e4b..9c581df9a 100644
--- a/docs/public-api/crawlee-types.api.md
+++ b/docs/public-api/crawlee-types.api.md
@@ -440,6 +440,7 @@ export type SearchParams = string | URLSearchParams | Record<string, string | nu
export interface SendRequestOptions {
// (undocumented)
cookieJar?: CookieJar;
+ ignoreTlsErrors?: boolean;
proxyUrl?: string;
// (undocumented)
session?: ISession;
diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.md
index e8fbf8540..7e720872e 100644
--- a/docs/upgrading/upgrading_v4.md
+++ b/docs/upgrading/upgrading_v4.md
@@ -401,6 +401,14 @@ Previously, `BrowserCrawler` with `saveResponseCookies` (formerly `persistCookie
In v4, when `saveResponseCookies` is enabled (the default), browser cookies are also re-read and stored in the session cookie jar **after** `requestHandler` completes. If you relied on handler-set cookies staying page-local and not affecting later requests on the same session, set `saveResponseCookies: false` or clear/overwrite cookies on the session explicitly.
+### `ignoreSslErrors` is renamed to `ignoreTlsErrors`
+
+The crawler option is renamed to `ignoreTlsErrors`, matching the naming used everywhere else in v4 (`session.proxyInfo.ignoreTlsErrors`, the browser pool, the impit client). The old `ignoreSslErrors` name is no longer accepted — rename it in your crawler options. Behavior is unchanged from v3: the option defaults to `true` and HTTP crawlers accept invalid TLS certificates by default.
+
+Under the hood the crawler now forwards the option to the HTTP client as `SendRequestOptions.ignoreTlsErrors` on every navigation request, and the same flag is enabled automatically for MITM proxy sessions (`session.proxyInfo.ignoreTlsErrors`), matching the browser crawlers.
+
+This only affects custom `BaseHttpClient` implementations: honor `ignoreTlsErrors` (from `SendRequestOptions`, or `CustomFetchOptions` when extending the `BaseHttpClient` class from `@crawlee/http-client`) if your client can disable TLS verification. The built-in impit and got-scraping clients do; the native fetch fallback cannot, so it warns and ignores the flag.
+
### `preNavigationHooks` in `HttpCrawler` no longer accepts `gotOptions` object
The `preNavigationHooks` option in `HttpCrawler` subclasses no longer accepts the `gotOptions` object as a second parameter. Modify the `crawlingContext` fields (e.g. `.request`) directly instead.
@@ -685,7 +693,7 @@ This is intentional: these were never a supported API. If you relied on overridi
The change spans, among others:
- **`BasicCrawler`** — `unexpectedStop`, `requestHandlerTimeoutMillis`, `sameDomainDelayMillis`, `domainAccessedTime`, `handledRequestsCount`, `statusMessageLoggingInterval`, `statusMessageCallback`, `ignoreHttpErrorStatusCodes`, `taskLoopOptions` (was `autoscaledPoolOptions`), `autoscaledPool`, `respectRobotsTxtFile`, and the helpers `buildBasicContextPipeline`, `validateRequestUserData`, `pauseOnMigration`, `fetchNextRequest`, `delayRequest`, `handleRequest`, `timeoutAndRetry`, `isTaskReadyFunction`, `defaultIsFinishedFunction`, `requestFunctionErrorHandler`, `handleFailedRequestHandler`, `canRequestBeRetried`
-- **`HttpCrawler`** — `preNavigationHooks`, `postNavigationHooks`, `saveResponseCookies`, `navigationTimeoutMillis`, `ignoreSslErrors`, `suggestResponseEncoding`, `forceResponseEncoding`, `supportedMimeTypes`, and the helpers `requestFunction`, `parseResponse`, `getRequestOptions`, `encodeResponse`, `extendSupportedMimeTypes`, `handleRequestTimeout`
+- **`HttpCrawler`** — `preNavigationHooks`, `postNavigationHooks`, `saveResponseCookies`, `navigationTimeoutMillis`, `suggestResponseEncoding`, `forceResponseEncoding`, `supportedMimeTypes`, and the helpers `requestFunction`, `parseResponse`, `getRequestOptions`, `encodeResponse`, `extendSupportedMimeTypes`, `handleRequestTimeout`
- **`AutoscaledPool`** — the whole class is `@internal` in v4, so its members are not enumerated here; see [`AutoscaledPool` is no longer public API](#autoscaledpool-is-no-longer-public-api)
- **`SessionPool`** — all pool internals (`log`, `maxPoolSize`, `createSessionFunction`, `keyValueStore`, `sessions`, `sessionMap`, `sessionOptions`, `persistStateKey`, `persistStateKeyValueStoreId`, `events`, `persistenceOptions`, `sessionReuseStrategy`, and the helpers `ensureInitialized`, `maybeLoadSessionPool`, `registerSession`, `createSession`, `hasSpaceForSession`, `pickSession`, `removeRetiredSessions`, `getRandomIndex`, `defaultCreateSessionFunction`)
- **`Session`** — `maybeSelfRetire` (`userData` is now `readonly`)
diff --git a/packages/basic-crawler/src/internals/send-request.ts b/packages/basic-crawler/src/internals/send-request.ts
index 629c26451..850426447 100644
--- a/packages/basic-crawler/src/internals/send-request.ts
+++ b/packages/basic-crawler/src/internals/send-request.ts
@@ -36,6 +36,7 @@ export function createSendRequest(httpClient: BaseHttpClient, originRequest: Cra
cookieJar: overrideOptions?.cookieJar ?? session.cookieJar,
timeoutMillis: overrideOptions.timeoutMillis,
signal: overrideOptions.signal,
+ ignoreTlsErrors: overrideOptions.ignoreTlsErrors,
});
};
}
diff --git a/packages/got-scraping-client/src/index.ts b/packages/got-scraping-client/src/index.ts
index 298c47024..a267f6de5 100644
--- a/packages/got-scraping-client/src/index.ts
+++ b/packages/got-scraping-client/src/index.ts
@@ -35,7 +35,7 @@ export class GotScrapingHttpClient extends BaseHttpClient {
}
override async fetch(request: Request, options?: RequestInit & CustomFetchOptions): Promise<Response> {
- const { proxyUrl, redirect } = options ?? {};
+ const { proxyUrl, redirect, ignoreTlsErrors } = options ?? {};
if (!this.validateRequest(request)) {
throw new Error(`The HTTP method CONNECT is not supported by the GotScrapingHttpClient.`);
@@ -49,6 +49,7 @@ export class GotScrapingHttpClient extends BaseHttpClient {
proxyUrl,
signal: options?.signal ?? undefined,
followRedirect: redirect === 'follow',
+ ...(ignoreTlsErrors ? { https: { rejectUnauthorized: false } } : {}),
});
const responseHeaders = this.parseHeaders(gotResult.headers);
diff --git a/packages/http-client/src/base-http-client.ts b/packages/http-client/src/base-http-client.ts
index fe5b388d6..9b13ad047 100644
--- a/packages/http-client/src/base-http-client.ts
+++ b/packages/http-client/src/base-http-client.ts
@@ -33,6 +33,15 @@ export interface CustomFetchOptions {
* rest on a best-effort basis. Sourced from `SendRequestOptions.session.fingerprint`.
*/
fingerprint?: SessionFingerprint;
+
+ /**
+ * When `true`, TLS certificate errors should be ignored for this request.
+ * Set when `SendRequestOptions.ignoreTlsErrors` is passed (e.g. from the
+ * `ignoreTlsErrors` crawler option) or when the session's proxy is a MITM
+ * proxy (`session.proxyInfo.ignoreTlsErrors`). Best-effort: clients that
+ * cannot disable TLS verification ignore it.
+ */
+ ignoreTlsErrors?: boolean;
}
/**
@@ -105,6 +114,7 @@ export abstract class BaseHttpClient implements BaseHttpClientInterface {
cookieJar: CookieJar;
signal?: AbortSignal;
fingerprint?: SessionFingerprint;
+ ignoreTlsErrors?: boolean;
}> {
const proxyUrl = options?.proxyUrl ?? options?.session?.proxyInfo?.url;
const cookieJar = options?.cookieJar ?? options?.session?.cookieJar ?? (await this.#createDefaultCookieJar());
@@ -114,6 +124,7 @@ export abstract class BaseHttpClient implements BaseHttpClientInterface {
cookieJar,
signal,
fingerprint: options?.session?.fingerprint,
+ ignoreTlsErrors: options?.ignoreTlsErrors || options?.session?.proxyInfo?.ignoreTlsErrors,
};
}
@@ -176,7 +187,7 @@ export abstract class BaseHttpClient implements BaseHttpClientInterface {
let currentRequest = initialRequest;
let redirectCount = 0;
- const { proxyUrl, cookieJar, signal, fingerprint } = await this.resolveRequestContext(options);
+ const { proxyUrl, cookieJar, signal, fingerprint, ignoreTlsErrors } = await this.resolveRequestContext(options);
currentRequest = initialRequest.clone();
while (true) {
@@ -187,6 +198,7 @@ export abstract class BaseHttpClient implements BaseHttpClientInterface {
proxyUrl,
cookieJar,
fingerprint,
+ ignoreTlsErrors,
redirect: 'manual',
});
diff --git a/packages/http-client/src/fetch-http-client.ts b/packages/http-client/src/fetch-http-client.ts
index 48395390f..f4be3c298 100644
--- a/packages/http-client/src/fetch-http-client.ts
+++ b/packages/http-client/src/fetch-http-client.ts
@@ -1,3 +1,5 @@
+import type { CrawleeLogger } from '@crawlee/types';
+
import { BaseHttpClient, type CustomFetchOptions } from './base-http-client.js';
/**
@@ -6,7 +8,21 @@ import { BaseHttpClient, type CustomFetchOptions } from './base-http-client.js';
* This implementation does not support proxying.
*/
export class FetchHttpClient extends BaseHttpClient {
+ #logger?: CrawleeLogger;
+
+ constructor(options?: { logger?: CrawleeLogger }) {
+ super(options);
+ this.#logger = options?.logger;
+ }
+
override async fetch(request: Request, options?: RequestInit & CustomFetchOptions): Promise<Response> {
+ if (options?.ignoreTlsErrors) {
+ this.#logger?.warningOnce(
+ 'FetchHttpClient cannot disable TLS certificate verification, the `ignoreTlsErrors` option is ignored. ' +
+ 'Install the optional @crawlee/impit-client dependency to make it work.',
+ );
+ }
+
return fetch(request, options);
}
}
diff --git a/packages/http-crawler/src/internals/http-crawler.ts b/packages/http-crawler/src/internals/http-crawler.ts
index e40ac25de..bb54f6600 100644
--- a/packages/http-crawler/src/internals/http-crawler.ts
+++ b/packages/http-crawler/src/internals/http-crawler.ts
@@ -89,9 +89,14 @@ export interface HttpCrawlerOptions<
navigationTimeoutSecs?: number;
/**
- * If set to true, SSL certificate errors will be ignored.
+ * If set to `true`, TLS/SSL certificate errors are ignored. Forwarded to the HTTP client as
+ * {@apilink SendRequestOptions.ignoreTlsErrors|`ignoreTlsErrors`} on every navigation request, so custom
+ * {@apilink BaseHttpClient} implementations should honor that flag (the built-in impit and got-scraping
+ * clients do; the native fetch fallback cannot disable TLS verification and warns instead).
+ *
+ * @default true
*/
- ignoreSslErrors?: boolean;
+ ignoreTlsErrors?: boolean;
/**
* Async functions that are sequentially evaluated before the navigation. Good for setting additional cookies
@@ -348,8 +353,7 @@ export class HttpCrawler<
#postNavigationHooks: InternalHttpPostNavigationHook[];
#saveResponseCookies: boolean;
#navigationTimeoutMillis: number;
- // kept as TS-private: tests read it at runtime
- private ignoreSslErrors: boolean;
+ #ignoreTlsErrors: boolean;
#suggestResponseEncoding?: string;
#forceResponseEncoding?: string;
readonly #supportedMimeTypes: Set<string>;
@@ -358,7 +362,7 @@ export class HttpCrawler<
...BasicCrawler.optionsShape,
navigationTimeoutSecs: ow.optional.number,
- ignoreSslErrors: ow.optional.boolean,
+ ignoreTlsErrors: ow.optional.boolean,
additionalMimeTypes: ow.optional.array.ofType(ow.string),
suggestResponseEncoding: ow.optional.string,
forceResponseEncoding: ow.optional.string,
@@ -379,7 +383,7 @@ export class HttpCrawler<
const {
navigationTimeoutSecs = 30,
- ignoreSslErrors = true,
+ ignoreTlsErrors = true,
additionalMimeTypes = [],
suggestResponseEncoding,
forceResponseEncoding,
@@ -409,7 +413,7 @@ export class HttpCrawler<
}
this.#navigationTimeoutMillis = navigationTimeoutSecs * 1000;
- this.ignoreSslErrors = ignoreSslErrors;
+ this.#ignoreTlsErrors = ignoreTlsErrors;
this.#suggestResponseEncoding = suggestResponseEncoding;
this.#forceResponseEncoding = forceResponseEncoding;
// Cast away the extension-aware option types to the base internal storage types (see the field
@@ -764,9 +768,6 @@ export class HttpCrawler<
timeout: this.#navigationTimeoutMillis,
sessionToken: session,
headers: request.headers,
- https: {
- rejectUnauthorized: !this.ignoreSslErrors,
- },
body: undefined as string | undefined,
};
@@ -775,14 +776,6 @@ export class HttpCrawler<
delete requestOptions.headers!.cookie;
}
- // Disable SSL verification for MITM proxies
- if (session.proxyInfo?.ignoreTlsErrors) {
- requestOptions.https = {
- ...requestOptions.https,
- rejectUnauthorized: false,
- };
- }
-
if (/PATCH|POST|PUT/.test(request.method)) requestOptions.body = request.payload ?? '';
return requestOptions;
@@ -911,6 +904,7 @@ export class HttpCrawler<
cookieJar,
signal: cancelSignal,
timeoutMillis: cancelSignal ? undefined : opts.timeout,
+ ignoreTlsErrors: this.#ignoreTlsErrors,
},
);
diff --git a/packages/impit-client/src/index.ts b/packages/impit-client/src/index.ts
index 9647a0fb2..89549869e 100644
--- a/packages/impit-client/src/index.ts
+++ b/packages/impit-client/src/index.ts
@@ -91,13 +91,16 @@ export class ImpitHttpClient extends BaseHttpClient {
* @inheritDoc
*/
async fetch(request: Request, options?: RequestInit & CustomFetchOptions): Promise<Response> {
- const { proxyUrl, redirect, signal, fingerprint } = options ?? {};
+ const { proxyUrl, redirect, signal, fingerprint, ignoreTlsErrors } = options ?? {};
const impitBrowser = this.resolveImpitBrowser(fingerprint);
const impit = this.getClient({
...this.#impitOptions,
...(impitBrowser ? { browser: impitBrowser } : {}),
+ // The per-request flag (from the crawler option or a MITM proxy session)
+ // can only enable ignoring, never override a constructor-level `true`.
+ ...(ignoreTlsErrors ? { ignoreTlsErrors: true } : {}),
proxyUrl,
followRedirects: redirect === 'follow',
});
diff --git a/packages/types/src/http-client.ts b/packages/types/src/http-client.ts
index b7e6e0d2f..f29a68a85 100644
--- a/packages/types/src/http-client.ts
+++ b/packages/types/src/http-client.ts
@@ -75,6 +75,13 @@ export interface SendRequestOptions {
* Note that setting this manually can interfere with session proxy rotation.
*/
proxyUrl?: string;
+
+ /**
+ * When `true`, TLS certificate errors are ignored for this request. Also enabled by
+ * `session.proxyInfo.ignoreTlsErrors` (MITM proxies). Best-effort: clients that cannot
+ * disable TLS verification (e.g. the native fetch fallback) ignore it.
+ */
+ ignoreTlsErrors?: boolean;
}
export interface StreamOptions extends SendRequestOptions {
diff --git a/packages/types/src/session.ts b/packages/types/src/session.ts
index 70618eff6..a0faa9438 100644
--- a/packages/types/src/session.ts
+++ b/packages/types/src/session.ts
@@ -55,6 +55,8 @@ export interface ProxyInfo {
/**
* When `true`, the proxy is likely intercepting HTTPS traffic and is able to view and modify its content.
+ * The built-in HTTP clients and the browser pool disable TLS certificate verification for the session's
+ * requests when this is set.
*
* @default false
*/
diff --git a/test/core/base-http-client.test.ts b/test/core/base-http-client.test.ts
index e956ab5b0..5e4084ffc 100644
--- a/test/core/base-http-client.test.ts
+++ b/test/core/base-http-client.test.ts
@@ -1,9 +1,10 @@
import http from 'node:http';
import type { AddressInfo } from 'node:net';
-import { FetchHttpClient } from '@crawlee/http-client';
+import type { CustomFetchOptions } from '@crawlee/http-client';
+import { BaseHttpClient, FetchHttpClient } from '@crawlee/http-client';
import { CookieJar } from 'tough-cookie';
-import { afterAll, beforeAll, describe, expect, test } from 'vitest';
+import { afterAll, beforeAll, describe, expect, test, vitest } from 'vitest';
let server: http.Server;
let url: string;
@@ -142,3 +143,60 @@ describe('BaseHttpClient cookie handling', () => {
expect(body.cookie).toBe('header_only=value');
});
});
+
+describe('BaseHttpClient TLS error handling', () => {
+ class CapturingHttpClient extends BaseHttpClient {
+ lastFetchOptions?: RequestInit & CustomFetchOptions;
+
+ override async fetch(request: Request, options?: RequestInit & CustomFetchOptions): Promise<Response> {
+ this.lastFetchOptions = options;
+ return fetch(request, options);
+ }
+ }
+
+ test('passes ignoreTlsErrors from session.proxyInfo to fetch', async () => {
+ const client = new CapturingHttpClient();
+ const session = { proxyInfo: { ignoreTlsErrors: true } } as any;
+
+ await client.sendRequest(new Request(url), { session });
+
+ expect(client.lastFetchOptions?.ignoreTlsErrors).toBe(true);
+ });
+
+ test('passes an explicit ignoreTlsErrors option to fetch', async () => {
+ const client = new CapturingHttpClient();
+
+ await client.sendRequest(new Request(url), { ignoreTlsErrors: true });
+
+ expect(client.lastFetchOptions?.ignoreTlsErrors).toBe(true);
+ });
+
+ test('session proxyInfo.ignoreTlsErrors wins over an explicit false option', async () => {
+ const client = new CapturingHttpClient();
+ const session = { proxyInfo: { ignoreTlsErrors: true } } as any;
+
+ await client.sendRequest(new Request(url), { session, ignoreTlsErrors: false });
+
+ expect(client.lastFetchOptions?.ignoreTlsErrors).toBe(true);
+ });
+
+ test('leaves ignoreTlsErrors unset without a session proxy', async () => {
+ const client = new CapturingHttpClient();
+
+ await client.sendRequest(new Request(url));
+
+ expect(client.lastFetchOptions?.ignoreTlsErrors).toBeUndefined();
+ });
+
+ test('FetchHttpClient warns when ignoreTlsErrors is requested', async () => {
+ const warningOnce = vitest.fn();
+ const client = new FetchHttpClient({ logger: { warningOnce } as any });
+
+ await client.sendRequest(new Request(url), { ignoreTlsErrors: true });
+ expect(warningOnce).toHaveBeenCalledWith(expect.stringContaining('ignoreTlsErrors'));
+
+ warningOnce.mockClear();
+ await client.sendRequest(new Request(url));
+ expect(warningOnce).not.toHaveBeenCalled();
+ });
+});
diff --git a/test/core/crawlers/basic_crawler.test.ts b/test/core/crawlers/basic_crawler.test.ts
index 9c3863c06..1ef9b1a25 100644
--- a/test/core/crawlers/basic_crawler.test.ts
+++ b/test/core/crawlers/basic_crawler.test.ts
@@ -2326,6 +2326,26 @@ describe('BasicCrawler', () => {
]);
});
+ test('forwards ignoreTlsErrors to the http client', async () => {
+ const captured: (boolean | undefined)[] = [];
+
+ const crawler = new BasicCrawler({
+ httpClient: {
+ async sendRequest(request, options) {
+ captured.push(options?.ignoreTlsErrors);
+ return new Response('ok');
+ },
+ },
+ async requestHandler({ sendRequest }) {
+ await sendRequest({}, { ignoreTlsErrors: true });
+ },
+ });
+
+ await crawler.run([url]);
+
+ expect(captured).toEqual([true]);
+ });
+
test('proxyUrl TypeScript support', async () => {
const crawler = new BasicCrawler({
async requestHandler({ sendRequest }) {
diff --git a/test/core/crawlers/cheerio_crawler.test.ts b/test/core/crawlers/cheerio_crawler.test.ts
index d019c7b29..cfddf75a8 100644
--- a/test/core/crawlers/cheerio_crawler.test.ts
+++ b/test/core/crawlers/cheerio_crawler.test.ts
@@ -201,21 +201,32 @@ describe('CheerioCrawler', () => {
);
});
- test('should ignore ssl by default', async () => {
- const sources = [{ url: 'http://example.com/?q=1' }];
- const requestList = await RequestList.open(null, sources);
- const requestHandler = () => {};
+ test('forwards ignoreTlsErrors to the http client', async () => {
+ const captured: (boolean | undefined)[] = [];
+ const fetchClient = new FetchHttpClient();
+ const capturingClient = {
+ sendRequest: async (request: globalThis.Request, options?: { ignoreTlsErrors?: boolean }) => {
+ captured.push(options?.ignoreTlsErrors);
+ return fetchClient.sendRequest(request, options);
+ },
+ };
- const cheerioCrawler = new CheerioCrawler({
- requestList,
+ const defaultCrawler = new CheerioCrawler({
+ httpClient: capturingClient,
maxConcurrency: 1,
- requestHandler,
+ requestHandler: () => {},
});
+ await defaultCrawler.run([`${serverAddress}/?tls=default`]);
- await cheerioCrawler.run();
+ const strictCrawler = new CheerioCrawler({
+ httpClient: capturingClient,
+ ignoreTlsErrors: false,
+ maxConcurrency: 1,
+ requestHandler: () => {},
+ });
+ await strictCrawler.run([`${serverAddress}/?tls=strict`]);
- // @ts-expect-error Accessing private prop
- expect(cheerioCrawler.ignoreSslErrors).toBeTruthy();
+ expect(captured).toEqual([true, false]);
});
test('should work with skipNavigation', async () => {
diff --git a/test/core/got_scraping_http_client.test.ts b/test/core/got_scraping_http_client.test.ts
new file mode 100644
index 000000000..4532cbb19
--- /dev/null
+++ b/test/core/got_scraping_http_client.test.ts
@@ -0,0 +1,34 @@
+import { GotScrapingHttpClient } from '@crawlee/got-scraping-client';
+import { gotScraping } from 'got-scraping';
+
+vi.mock('got-scraping', () => ({
+ gotScraping: vi.fn(async () => ({
+ rawBody: Buffer.from('ok'),
+ headers: { 'content-type': 'text/plain' },
+ statusCode: 200,
+ statusMessage: 'OK',
+ url: 'http://example.com/',
+ })),
+}));
+
+describe('GotScrapingHttpClient', () => {
+ beforeEach(() => {
+ vi.mocked(gotScraping).mockClear();
+ });
+
+ test('disables certificate verification when ignoreTlsErrors is set', async () => {
+ const httpClient = new GotScrapingHttpClient();
+
+ await httpClient.fetch(new Request('http://example.com'), { ignoreTlsErrors: true });
+
+ expect(gotScraping).toHaveBeenCalledWith(expect.objectContaining({ https: { rejectUnauthorized: false } }));
+ });
+
+ test('leaves certificate verification enabled without the flag', async () => {
+ const httpClient = new GotScrapingHttpClient();
+
+ await httpClient.fetch(new Request('http://example.com'), {});
+
+ expect(gotScraping).toHaveBeenCalledWith(expect.not.objectContaining({ https: expect.anything() }));
+ });
+});
diff --git a/test/core/impit_http_client.test.ts b/test/core/impit_http_client.test.ts
index 4f7542186..58aa76481 100644
--- a/test/core/impit_http_client.test.ts
+++ b/test/core/impit_http_client.test.ts
@@ -4,42 +4,11 @@ import { Impit } from 'impit';
vi.mock('impit', () => ({
Impit: vi.fn(
class {
- fetch = vi.fn();
+ fetch = vi.fn(async () => new Response('ok'));
},
),
}));
-function createRedirectResponse(status: number, location: string, setCookie: string[] = []) {
- const headers = new Headers({ location });
- for (const cookie of setCookie) headers.append('set-cookie', cookie);
-
- return {
- status,
- statusText: 'Found',
- url: 'http://example.com/start',
- headers,
- body: undefined,
- };
-}
-
-function createFinalResponse(body = 'ok') {
- return {
- status: 200,
- statusText: 'OK',
- url: 'http://example.com/final',
- headers: new Headers({ 'content-type': 'text/plain', 'content-length': String(body.length) }),
- body: new ReadableStream({
- start(controller) {
- controller.enqueue(new TextEncoder().encode(body));
- controller.close();
- },
- }),
- text: async () => body,
- json: async () => ({ body }),
- bytes: async () => Buffer.from(body),
- };
-}
-
describe('ImpitHttpClient', () => {
beforeEach(() => {
vi.mocked(Impit).mockClear();
@@ -63,69 +32,19 @@ describe('ImpitHttpClient', () => {
expect(Impit).toHaveBeenCalledTimes(2);
});
- test('stream() invokes onRedirect and forwards mutated Cookie header to the next hop', async () => {
- const httpClient = new ImpitHttpClient({ cacheClients: false });
- const fetchMock = vi.fn();
-
- vi.mocked(Impit).mockImplementation(
- class {
- fetch = fetchMock;
- } as any,
- );
-
- fetchMock
- .mockResolvedValueOnce(createRedirectResponse(302, '/final', ['session=abc', 'other=def']))
- .mockResolvedValueOnce(createFinalResponse('done'));
-
- const onRedirect = vi.fn((_redirectResponse, updatedRequest) => {
- updatedRequest.headers.Cookie = 'session=abc';
- });
-
- const response = await httpClient.stream(
- {
- url: 'http://example.com/start',
- method: 'GET',
- headers: {},
- },
- onRedirect,
- );
-
- expect(onRedirect).toHaveBeenCalledTimes(1);
- expect(fetchMock).toHaveBeenCalledTimes(2);
+ test('forwards the per-request ignoreTlsErrors flag to the impit client', async () => {
+ const httpClient = new ImpitHttpClient();
- const [redirectResponse] = onRedirect.mock.calls[0];
- expect(redirectResponse.statusCode).toBe(302);
- expect(redirectResponse.headers['set-cookie']).toEqual(['session=abc', 'other=def']);
+ await httpClient.fetch(new Request('http://example.com'), { ignoreTlsErrors: true });
- const secondCallHeaders = fetchMock.mock.calls[1][1].headers as Headers;
- expect(secondCallHeaders.get('Cookie')).toBe('session=abc');
- expect(fetchMock.mock.calls[1][0]).toBe('http://example.com/final');
- expect(response.statusCode).toBe(200);
- expect(response.redirectUrls).toEqual([new URL('http://example.com/final')]);
+ expect(Impit).toHaveBeenCalledWith(expect.objectContaining({ ignoreTlsErrors: true }));
});
- test('stream() follows redirects without onRedirect for API compatibility', async () => {
- const httpClient = new ImpitHttpClient({ cacheClients: false });
- const fetchMock = vi.fn();
-
- vi.mocked(Impit).mockImplementation(
- class {
- fetch = fetchMock;
- } as any,
- );
-
- fetchMock
- .mockResolvedValueOnce(createRedirectResponse(302, 'http://example.com/final'))
- .mockResolvedValueOnce(createFinalResponse('done'));
+ test('keeps constructor-level ignoreTlsErrors when the per-request flag is absent', async () => {
+ const httpClient = new ImpitHttpClient({ ignoreTlsErrors: true });
- const response = await httpClient.stream({
- url: 'http://example.com/start',
- method: 'GET',
- headers: {},
- });
+ await httpClient.fetch(new Request('http://example.com'), {});
- expect(fetchMock).toHaveBeenCalledTimes(2);
- expect(response.statusCode).toBe(200);
- expect(response.redirectUrls).toHaveLength(1);
+ expect(Impit).toHaveBeenCalledWith(expect.objectContaining({ ignoreTlsErrors: true }));
});
});
diff --git a/test/e2e/cheerio-ignore-ssl-errors/actor/main.js b/test/e2e/cheerio-ignore-ssl-errors/actor/main.js
index f38f23df4..c96f6347d 100644
--- a/test/e2e/cheerio-ignore-ssl-errors/actor/main.js
+++ b/test/e2e/cheerio-ignore-ssl-errors/actor/main.js
@@ -11,7 +11,7 @@ const mainOptions = {
await Actor.main(async () => {
const crawler = new CheerioCrawler({
- ignoreSslErrors: true,
+ ignoreTlsErrors: true,
async requestHandler({ $, enqueueLinks, request, log }) {
const {
url,
diff --git a/test/e2e/cheerio-throw-on-ssl-errors/actor/main.js b/test/e2e/cheerio-throw-on-ssl-errors/actor/main.js
index 5897199d8..b926ccbcb 100644
--- a/test/e2e/cheerio-throw-on-ssl-errors/actor/main.js
+++ b/test/e2e/cheerio-throw-on-ssl-errors/actor/main.js
@@ -11,7 +11,7 @@ const mainOptions = {
await Actor.main(async () => {
const crawler = new CheerioCrawler({
- ignoreSslErrors: false,
+ ignoreTlsErrors: false,
async requestHandler({ $, enqueueLinks, request, log }) {
const {
userData: { label },