Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
docs: add guide for Impit HTTP client (#3359)
Martin Adámekd70fc8f732b8
diff --git a/docs/guides/impit-http-client/advanced-config.ts b/docs/guides/impit-http-client/advanced-config.tsnew file mode 100644index 000000000..c94af6516--- /dev/null+++ b/docs/guides/impit-http-client/advanced-config.ts@@ -0,0 +1,16 @@+import { CheerioCrawler } from 'crawlee';+import { ImpitHttpClient, Browser } from '@crawlee/impit-client';++const crawler = new CheerioCrawler({+ httpClient: new ImpitHttpClient({+ // Impersonate Chrome browser+ browser: Browser.Chrome,+ // Enable HTTP/3 protocol+ http3: true,+ }),+ async requestHandler({ $ }) {+ console.log(`Title: ${$('title').text()}`);+ },+});++await crawler.run(['https://example.com']);diff --git a/docs/guides/impit-http-client/basic-usage.ts b/docs/guides/impit-http-client/basic-usage.tsnew file mode 100644index 000000000..1a8754c9f--- /dev/null+++ b/docs/guides/impit-http-client/basic-usage.ts@@ -0,0 +1,14 @@+import { BasicCrawler } from 'crawlee';+import { ImpitHttpClient, Browser } from '@crawlee/impit-client';++const crawler = new BasicCrawler({+ httpClient: new ImpitHttpClient({+ browser: Browser.Firefox,+ }),+ async requestHandler({ sendRequest, log }) {+ const response = await sendRequest();+ log.info('Received response', { statusCode: response.statusCode });+ },+});++await crawler.run(['https://example.com']);diff --git a/docs/guides/impit-http-client/cheerio-crawler.ts b/docs/guides/impit-http-client/cheerio-crawler.tsnew file mode 100644index 000000000..0846cb834--- /dev/null+++ b/docs/guides/impit-http-client/cheerio-crawler.ts@@ -0,0 +1,23 @@+import { CheerioCrawler } from 'crawlee';+import { ImpitHttpClient, Browser } from '@crawlee/impit-client';++const crawler = new CheerioCrawler({+ httpClient: new ImpitHttpClient({+ browser: Browser.Chrome,+ }),+ async requestHandler({ $, request, enqueueLinks, pushData }) {+ const title = $('title').text();+ const h1 = $('h1').first().text();++ await pushData({+ url: request.url,+ title,+ h1,+ });++ // Enqueue links found on the page+ await enqueueLinks();+ },+});++await crawler.run(['https://example.com']);diff --git a/docs/guides/impit-http-client/http-crawler.ts b/docs/guides/impit-http-client/http-crawler.tsnew file mode 100644index 000000000..2d7fe34d2--- /dev/null+++ b/docs/guides/impit-http-client/http-crawler.ts@@ -0,0 +1,20 @@+import { HttpCrawler } from 'crawlee';+import { ImpitHttpClient, Browser } from '@crawlee/impit-client';++const crawler = new HttpCrawler({+ httpClient: new ImpitHttpClient({+ browser: Browser.Firefox,+ http3: true,+ }),+ async requestHandler({ body, request, log, pushData }) {+ log.info(`Processing ${request.url}`);++ // body is the raw HTML string+ await pushData({+ url: request.url,+ bodyLength: body.length,+ });+ },+});++await crawler.run(['https://example.com']);diff --git a/docs/guides/impit-http-client/impit-http-client.mdx b/docs/guides/impit-http-client/impit-http-client.mdxnew file mode 100644index 000000000..533d8dfd3--- /dev/null+++ b/docs/guides/impit-http-client/impit-http-client.mdx@@ -0,0 +1,139 @@+---+id: impit-http-client+title: Impit HTTP Client+description: Browser impersonation for HTTP requests using the Impit library+---++import CodeBlock from '@theme/CodeBlock';++import BasicUsageSource from '!!raw-loader!./basic-usage.ts';+import CheerioCrawlerSource from '!!raw-loader!./cheerio-crawler.ts';+import HttpCrawlerSource from '!!raw-loader!./http-crawler.ts';+import AdvancedConfigSource from '!!raw-loader!./advanced-config.ts';++## Introduction++The `ImpitHttpClient` is an HTTP client implementation based on the [Impit](https://github.com/apify/impit) library. It enables browser impersonation for HTTP requests, helping you bypass bot detection systems without running an actual browser.++:::info Successor to got-scraping++Impit is the successor to `got-scraping`, which is no longer actively maintained. We recommend using `ImpitHttpClient` for all new projects. Impit provides better anti-bot evasion through TLS fingerprinting and HTTP/3 support, while maintaining a smaller package size.++**Impit will become the default HTTP client in the next major version of Crawlee.**++:::++### Why use Impit?++Websites increasingly use sophisticated bot detection that analyzes:++- **HTTP fingerprints**: User-Agent strings, header ordering, HTTP/2 pseudo-header sequences+- **TLS fingerprints**: Cipher suites, TLS extensions, and cryptographic details in the ClientHello message++Standard HTTP clients like `fetch` or `axios` are easily detected because their fingerprints don't match real browsers. Unlike `got-scraping` which only handles HTTP-level fingerprinting, Impit also mimics TLS fingerprints, making requests appear to come from real browsers.++## Installation++Install the `@crawlee/impit-client` package:++```bash npm2yarn+npm install @crawlee/impit-client+```++:::note++The `impit` package includes native binaries and supports Windows, macOS (including ARM), and Linux out of the box.++:::++## Basic usage++Pass the `ImpitHttpClient` instance to the `httpClient` option of any Crawlee crawler:++<CodeBlock language="ts">{BasicUsageSource}</CodeBlock>++## Usage with different crawlers++### CheerioCrawler++<CodeBlock language="ts">{CheerioCrawlerSource}</CodeBlock>++### HttpCrawler++<CodeBlock language="ts">{HttpCrawlerSource}</CodeBlock>++## Configuration options++The `ImpitHttpClient` constructor accepts the following options:++| Option | Type | Default | Description |+|--------|------|---------|-------------|+| `browser` | `'chrome'` \| `'firefox'` | `undefined` | Browser to impersonate. Affects TLS fingerprint and default headers. |+| `http3` | `boolean` | `false` | Enable HTTP/3 (QUIC) protocol support. |+| `ignoreTlsErrors` | `boolean` | `false` | Ignore TLS certificate errors. Useful for testing or self-signed certificates. |++### Browser impersonation++Use the `Browser` enum to specify which browser to impersonate:++```ts+import { ImpitHttpClient, Browser } from '@crawlee/impit-client';++// Impersonate Firefox+const firefoxClient = new ImpitHttpClient({ browser: Browser.Firefox });++// Impersonate Chrome+const chromeClient = new ImpitHttpClient({ browser: Browser.Chrome });+```++### Advanced configuration++<CodeBlock language="ts">{AdvancedConfigSource}</CodeBlock>++## Proxy support++Proxies are configured per-request through Crawlee's proxy management system, not on the `ImpitHttpClient` itself. Use `ProxyConfiguration` as you normally would:++```ts+import { CheerioCrawler, ProxyConfiguration } from 'crawlee';+import { ImpitHttpClient, Browser } from '@crawlee/impit-client';++const proxyConfiguration = new ProxyConfiguration({+ proxyUrls: ['http://proxy1.example.com:8080', 'http://proxy2.example.com:8080'],+});++const crawler = new CheerioCrawler({+ httpClient: new ImpitHttpClient({ browser: Browser.Chrome }),+ proxyConfiguration,+ async requestHandler({ $, request }) {+ console.log(`Scraped ${request.url}`);+ },+});+```++## How it works++Impit achieves browser impersonation at two levels:++1. **HTTP level**: Mimics browser-specific header ordering, HTTP/2 settings, and pseudo-header sequences that antibot services analyze.++2. **TLS level**: Uses a patched version of `rustls` to replicate the exact TLS ClientHello message that browsers send, including cipher suites and extensions.++This dual-layer approach makes requests appear to come from a real browser, significantly reducing blocks from bot detection systems.++## Comparison with other solutions++| Feature | got-scraping | curl-impersonate | Impit |+|---------|--------------|------------------|-------|+| TLS fingerprinting | No | Yes | Yes |+| HTTP/3 support | No | Yes | Yes |+| Native Node.js package | Yes | No (child process) | Yes |+| Windows/macOS ARM | Yes | No | Yes |+| Package size | ~10 MB | ~20 MB | ~8 MB |++**Related links**++- [Impit GitHub repository](https://github.com/apify/impit)+- [Custom HTTP Client guide](./custom-http-client)+- [Proxy Management guide](./proxy-management)+- [Avoiding blocking guide](./avoid-blocking)diff --git a/website/sidebars.js b/website/sidebars.jsindex a4bfb9b9e..bc321b69c 100644--- a/website/sidebars.js+++ b/website/sidebars.js@@ -42,6 +42,7 @@ module.exports = {'guides/scaling-crawlers','guides/avoid-blocking','guides/jsdom-crawler-guide',+ 'guides/impit-http-client/impit-http-client','guides/got-scraping','guides/typescript-project','guides/docker-images',diff --git a/website/versioned_docs/version-3.14/guides/impit-http-client/advanced-config.ts b/website/versioned_docs/version-3.14/guides/impit-http-client/advanced-config.tsnew file mode 100644index 000000000..c94af6516--- /dev/null+++ b/website/versioned_docs/version-3.14/guides/impit-http-client/advanced-config.ts@@ -0,0 +1,16 @@+import { CheerioCrawler } from 'crawlee';+import { ImpitHttpClient, Browser } from '@crawlee/impit-client';++const crawler = new CheerioCrawler({+ httpClient: new ImpitHttpClient({+ // Impersonate Chrome browser+ browser: Browser.Chrome,+ // Enable HTTP/3 protocol+ http3: true,+ }),+ async requestHandler({ $ }) {+ console.log(`Title: ${$('title').text()}`);+ },+});++await crawler.run(['https://example.com']);diff --git a/website/versioned_docs/version-3.14/guides/impit-http-client/basic-usage.ts b/website/versioned_docs/version-3.14/guides/impit-http-client/basic-usage.tsnew file mode 100644index 000000000..1a8754c9f--- /dev/null+++ b/website/versioned_docs/version-3.14/guides/impit-http-client/basic-usage.ts@@ -0,0 +1,14 @@+import { BasicCrawler } from 'crawlee';+import { ImpitHttpClient, Browser } from '@crawlee/impit-client';++const crawler = new BasicCrawler({+ httpClient: new ImpitHttpClient({+ browser: Browser.Firefox,+ }),+ async requestHandler({ sendRequest, log }) {+ const response = await sendRequest();+ log.info('Received response', { statusCode: response.statusCode });+ },+});++await crawler.run(['https://example.com']);diff --git a/website/versioned_docs/version-3.14/guides/impit-http-client/cheerio-crawler.ts b/website/versioned_docs/version-3.14/guides/impit-http-client/cheerio-crawler.tsnew file mode 100644index 000000000..0846cb834--- /dev/null+++ b/website/versioned_docs/version-3.14/guides/impit-http-client/cheerio-crawler.ts@@ -0,0 +1,23 @@+import { CheerioCrawler } from 'crawlee';+import { ImpitHttpClient, Browser } from '@crawlee/impit-client';++const crawler = new CheerioCrawler({+ httpClient: new ImpitHttpClient({+ browser: Browser.Chrome,+ }),+ async requestHandler({ $, request, enqueueLinks, pushData }) {+ const title = $('title').text();+ const h1 = $('h1').first().text();++ await pushData({+ url: request.url,+ title,+ h1,+ });++ // Enqueue links found on the page+ await enqueueLinks();+ },+});++await crawler.run(['https://example.com']);diff --git a/website/versioned_docs/version-3.14/guides/impit-http-client/http-crawler.ts b/website/versioned_docs/version-3.14/guides/impit-http-client/http-crawler.tsnew file mode 100644index 000000000..2d7fe34d2--- /dev/null+++ b/website/versioned_docs/version-3.14/guides/impit-http-client/http-crawler.ts@@ -0,0 +1,20 @@+import { HttpCrawler } from 'crawlee';+import { ImpitHttpClient, Browser } from '@crawlee/impit-client';++const crawler = new HttpCrawler({+ httpClient: new ImpitHttpClient({+ browser: Browser.Firefox,+ http3: true,+ }),+ async requestHandler({ body, request, log, pushData }) {+ log.info(`Processing ${request.url}`);++ // body is the raw HTML string+ await pushData({+ url: request.url,+ bodyLength: body.length,+ });+ },+});++await crawler.run(['https://example.com']);diff --git a/website/versioned_docs/version-3.14/guides/impit-http-client/impit-http-client.mdx b/website/versioned_docs/version-3.14/guides/impit-http-client/impit-http-client.mdxnew file mode 100644index 000000000..533d8dfd3--- /dev/null+++ b/website/versioned_docs/version-3.14/guides/impit-http-client/impit-http-client.mdx@@ -0,0 +1,139 @@+---+id: impit-http-client+title: Impit HTTP Client+description: Browser impersonation for HTTP requests using the Impit library+---++import CodeBlock from '@theme/CodeBlock';++import BasicUsageSource from '!!raw-loader!./basic-usage.ts';+import CheerioCrawlerSource from '!!raw-loader!./cheerio-crawler.ts';+import HttpCrawlerSource from '!!raw-loader!./http-crawler.ts';+import AdvancedConfigSource from '!!raw-loader!./advanced-config.ts';++## Introduction++The `ImpitHttpClient` is an HTTP client implementation based on the [Impit](https://github.com/apify/impit) library. It enables browser impersonation for HTTP requests, helping you bypass bot detection systems without running an actual browser.++:::info Successor to got-scraping++Impit is the successor to `got-scraping`, which is no longer actively maintained. We recommend using `ImpitHttpClient` for all new projects. Impit provides better anti-bot evasion through TLS fingerprinting and HTTP/3 support, while maintaining a smaller package size.++**Impit will become the default HTTP client in the next major version of Crawlee.**++:::++### Why use Impit?++Websites increasingly use sophisticated bot detection that analyzes:++- **HTTP fingerprints**: User-Agent strings, header ordering, HTTP/2 pseudo-header sequences+- **TLS fingerprints**: Cipher suites, TLS extensions, and cryptographic details in the ClientHello message++Standard HTTP clients like `fetch` or `axios` are easily detected because their fingerprints don't match real browsers. Unlike `got-scraping` which only handles HTTP-level fingerprinting, Impit also mimics TLS fingerprints, making requests appear to come from real browsers.++## Installation++Install the `@crawlee/impit-client` package:++```bash npm2yarn+npm install @crawlee/impit-client+```++:::note++The `impit` package includes native binaries and supports Windows, macOS (including ARM), and Linux out of the box.++:::++## Basic usage++Pass the `ImpitHttpClient` instance to the `httpClient` option of any Crawlee crawler:++<CodeBlock language="ts">{BasicUsageSource}</CodeBlock>++## Usage with different crawlers++### CheerioCrawler++<CodeBlock language="ts">{CheerioCrawlerSource}</CodeBlock>++### HttpCrawler++<CodeBlock language="ts">{HttpCrawlerSource}</CodeBlock>++## Configuration options++The `ImpitHttpClient` constructor accepts the following options:++| Option | Type | Default | Description |+|--------|------|---------|-------------|+| `browser` | `'chrome'` \| `'firefox'` | `undefined` | Browser to impersonate. Affects TLS fingerprint and default headers. |+| `http3` | `boolean` | `false` | Enable HTTP/3 (QUIC) protocol support. |+| `ignoreTlsErrors` | `boolean` | `false` | Ignore TLS certificate errors. Useful for testing or self-signed certificates. |++### Browser impersonation++Use the `Browser` enum to specify which browser to impersonate:++```ts+import { ImpitHttpClient, Browser } from '@crawlee/impit-client';++// Impersonate Firefox+const firefoxClient = new ImpitHttpClient({ browser: Browser.Firefox });++// Impersonate Chrome+const chromeClient = new ImpitHttpClient({ browser: Browser.Chrome });+```++### Advanced configuration++<CodeBlock language="ts">{AdvancedConfigSource}</CodeBlock>++## Proxy support++Proxies are configured per-request through Crawlee's proxy management system, not on the `ImpitHttpClient` itself. Use `ProxyConfiguration` as you normally would:++```ts+import { CheerioCrawler, ProxyConfiguration } from 'crawlee';+import { ImpitHttpClient, Browser } from '@crawlee/impit-client';++const proxyConfiguration = new ProxyConfiguration({+ proxyUrls: ['http://proxy1.example.com:8080', 'http://proxy2.example.com:8080'],+});++const crawler = new CheerioCrawler({+ httpClient: new ImpitHttpClient({ browser: Browser.Chrome }),+ proxyConfiguration,+ async requestHandler({ $, request }) {+ console.log(`Scraped ${request.url}`);+ },+});+```++## How it works++Impit achieves browser impersonation at two levels:++1. **HTTP level**: Mimics browser-specific header ordering, HTTP/2 settings, and pseudo-header sequences that antibot services analyze.++2. **TLS level**: Uses a patched version of `rustls` to replicate the exact TLS ClientHello message that browsers send, including cipher suites and extensions.++This dual-layer approach makes requests appear to come from a real browser, significantly reducing blocks from bot detection systems.++## Comparison with other solutions++| Feature | got-scraping | curl-impersonate | Impit |+|---------|--------------|------------------|-------|+| TLS fingerprinting | No | Yes | Yes |+| HTTP/3 support | No | Yes | Yes |+| Native Node.js package | Yes | No (child process) | Yes |+| Windows/macOS ARM | Yes | No | Yes |+| Package size | ~10 MB | ~20 MB | ~8 MB |++**Related links**++- [Impit GitHub repository](https://github.com/apify/impit)+- [Custom HTTP Client guide](./custom-http-client)+- [Proxy Management guide](./proxy-management)+- [Avoiding blocking guide](./avoid-blocking)diff --git a/website/versioned_sidebars/version-3.14-sidebars.json b/website/versioned_sidebars/version-3.14-sidebars.jsonindex f7bd1d6ed..459f5f2a4 100644--- a/website/versioned_sidebars/version-3.14-sidebars.json+++ b/website/versioned_sidebars/version-3.14-sidebars.json@@ -43,6 +43,7 @@"guides/scaling-crawlers","guides/avoid-blocking","guides/jsdom-crawler-guide",+ "guides/impit-http-client/impit-http-client","guides/got-scraping","guides/typescript-project","guides/docker-images",