Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Update docker image guide [skip ci]
Ondra Urban7f401de83a1a
diff --git a/docs/guides/docker_images.md b/docs/guides/docker_images.mdindex 55dc02ffa..0d0288a08 100644--- a/docs/guides/docker_images.md+++ b/docs/guides/docker_images.md@@ -14,12 +14,12 @@ and in our [DockerHub](https://hub.docker.com/orgs/apify).Browsers are pretty big, so we try to provide a wide variety of images to suit your needs. Here's a full listof our Docker images.-- [`apify/actor-node-basic`](#actor-node-basic)-- [`apify/actor-node-chrome`](#actor-node-chrome)-- [`apify/actor-node-chrome-xvfb`](#actor-node-chrome-xvfb)+- [`apify/actor-node`](#actor-node)+- [`apify/actor-node-puppeteer-chrome`](#actor-node-puppeteer-chrome)- [`apify/actor-node-playwright`](#actor-node-playwright)- [`apify/actor-node-playwright-chrome`](#actor-node-playwright-chrome)- [`apify/actor-node-playwright-firefox`](#actor-node-playwright-firefox)+- [`apify/actor-node-playwright-webkit`](#actor-node-playwright-webkit)## Example DockerfileTo use our images, you need a [`Dockerfile`](https://docs.docker.com/engine/reference/builder/).@@ -30,7 +30,7 @@ which automatically copies the correct Dockerfile into your project folder.# First, specify the base Docker image. You can read more about# the available images at https://sdk.apify.com/docs/guides/docker-images# You can also use any other image from Docker Hub.-FROM apify/actor-node-basic+FROM apify/actor-node# Second, copy just package.json and package-lock.json since it should be# the only file that affects "npm install" in the next step, to speed up the build@@ -62,7 +62,54 @@ COPY . ./# CMD npm start```-## actor-node-basic+## Versioning+The images are tagged with the version of the library that's preinstalled in the image. This ensures+compatibility. For example, `apify/actor-node-puppeteer-chrome:5.5.0` comes with Puppeteer v5.5.0,+which bundles Chromium 88, and we add Chrome 88. If you try to install a different version of Puppeteer+into this image, you may run into compatibility issues.++Similarly `apify/actor-node-playwright-firefox:1.7.1` is pre-installed with the Firefox version that comes+with v1.7.1.++Installing `apify/actor-node-puppeteer-chrome` (without a tag) will install the latest available version.++We recommend reflecting this in your `package.json` files. Either by providing the same version of+the library as the selected image:++```dockerfile+FROM apify/actor-node-playwright-chrome:1.7.1+```++```json+{+ "dependencies": {+ "playwright": "1.7.1"+ }+}+```++Or by using an asterisk as your version. This will make sure the library version pre-installed in the+docker image is left untouched.++```json+{+ "dependencies": {+ "playwright": "*"+ }+}+```++## Warning about image size+Browsers are huge. If you don't need them all in your image, it's better to use a smaller image with+only the one browser you need.++Be careful when installing new dependencies. Nothing prevents you from installing Playwright into the+`actor-node-puppeteer-chrome` image, but the resulting image will be about 3 times larger and extremely+slow to download and build.++Use only what you need and you'll be rewarded with reasonable build and start times.++## actor-nodeThis is the smallest image we have based on Alpine Linux. It does not include any browsers, and it's thereforebest used with [`CheerioCrawler`](../api/cheerio-crawler). It benefits from lightning fast builds and container startups.@@ -70,31 +117,22 @@ best used with [`CheerioCrawler`](../api/cheerio-crawler). It benefits from lighand other browser based features will **NOT** work with this image.```dockerfile-FROM apify/actor-node-basic+FROM apify/actor-node```-## actor-node-chrome-This image includes Puppeteer (Chromium) and the `Chrome` browser. It can be used with+## actor-node-puppeteer-chrome+This image includes Puppeteer (Chromium) and the Chrome browser. It can be used with[`CheerioCrawler`](../api/cheerio-crawler) and [`PuppeteerCrawler`](../api/puppeteer-crawler), but **NOT** with-[`PlaywrightCrawler`](../api/playwright-crawler). If you prefer Puppeteer, you'll save some time and megabytes-by selecting this image.+[`PlaywrightCrawler`](../api/playwright-crawler).-It can only run `headless` browsers. For headful, see [`actor-node-chrome-xvfb`](#actor-node-chrome-xvfb).--```dockerfile-FROM apify/actor-node-chrome-```--## actor-node-chrome-xvfb-The [`actor-node-chrome`](#actor-node-chrome) image with XVFB installed (a display emulator). It allows-you to run browsers with `headful: false`.+The image supports XVFB by default, so you can run both `headless` and `headful` browsers with it.```dockerfile-FROM apify/actor-node-chrome-xvfb+FROM apify/actor-node-puppeteer-chrome```## actor-node-playwright-A very large and slow image that can run everything. Puppeteer, Playwright, Chromium, Chrome, Firefox,+A very large and slow image that can run all Playwright browsers: Chromium, Chrome, Firefox,WebKit. Everything is installed. If you need to develop or test with multiple browsers, this is the image to choose,but in most cases, we suggest using the specialized images below.@@ -103,15 +141,15 @@ FROM apify/actor-node-playwright```## actor-node-playwright-chrome-Similar to [`actor-node-chrome`](#actor-node-chrome), but for Playwright. You can run+Similar to [`actor-node-puppeteer-chrome`](#actor-node-puppeteer-chrome), but for Playwright. You can run[`CheerioCrawler`](../api/cheerio-crawler) and [`PlaywrightCrawler`](../api/playwright-crawler),but **NOT** [`PuppeteerCrawler`](../api/puppeteer-crawler).-> Technically, it should also work with Puppeteer if you install it, but it's not guaranteed.+> Technically, it should also work with Puppeteer if you install it yourself, but it's not guaranteed.It uses the [`PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD`](https://playwright.dev/docs/api/environment-variables/)environment variable to block installation of more browsers into your images (to keep them small).-If you want more browsers, either choose the [`actor-node-playwright](#actor-node-playwright) image+If you want more browsers, either choose the [`actor-node-playwright`](#actor-node-playwright) imageor override this env var.The image supports XVFB by default, so you can run both `headless` and `headful` browsers with it.diff --git a/website/i18n/en.json b/website/i18n/en.jsonindex 25001d176..839eb2743 100644--- a/website/i18n/en.json+++ b/website/i18n/en.json@@ -344,27 +344,12 @@"version-0.22.4/api/version-0.22.4-browser-crawler": {"title": "BrowserCrawler"},- "version-0.22.4/api/version-0.22.4-cache-container": {- "title": "CacheContainer"- },"version-0.22.4/api/version-0.22.4-cheerio-crawler": {"title": "CheerioCrawler"},- "version-0.22.4/api/version-0.22.4-cookie-parse-error": {- "title": "CookieParseError"- },"version-0.22.4/api/version-0.22.4-dataset": {"title": "Dataset"},- "version-0.22.4/api/version-0.22.4-exports": {- "title": "exports"- },- "version-0.22.4/api/version-0.22.4-global-cache": {- "title": "GlobalCache"- },- "version-0.22.4/api/version-0.22.4-job": {- "title": "Job"- },"version-0.22.4/api/version-0.22.4-key-value-store": {"title": "KeyValueStore"},@@ -416,9 +401,6 @@"version-0.22.4/api/version-0.22.4-statistics": {"title": "Statistics"},- "version-0.22.4/api/version-0.22.4-storage-manager": {- "title": "StorageManager"- },"version-0.22.4/api/version-0.22.4-system-status": {"title": "SystemStatus"},@@ -590,12 +572,6 @@"version-0.22.4/typedefs/version-0.22.4-memory-info": {"title": "MemoryInfo"},- "version-0.22.4/typedefs/version-0.22.4-playwright-crawler-options": {- "title": "PlaywrightCrawlerOptions"- },- "version-0.22.4/typedefs/version-0.22.4-playwright-launch-context": {- "title": "PlaywrightLaunchContext"- },"version-0.22.4/typedefs/version-0.22.4-post-response": {"title": "PostResponse"},diff --git a/website/versioned_docs/version-0.22.4/api/CacheContainer.md b/website/versioned_docs/version-0.22.4/api/CacheContainer.mddeleted file mode 100644index 2d328d61e..000000000--- a/website/versioned_docs/version-0.22.4/api/CacheContainer.md+++ /dev/null@@ -1,7 +0,0 @@-----id: version-0.22.4-cache-container-title: CacheContainer-original_id: cache-container------ERROR, Cannot find class.diff --git a/website/versioned_docs/version-0.22.4/api/CookieParseError.md b/website/versioned_docs/version-0.22.4/api/CookieParseError.mddeleted file mode 100644index 5b668afcb..000000000--- a/website/versioned_docs/version-0.22.4/api/CookieParseError.md+++ /dev/null@@ -1,19 +0,0 @@-----id: version-0.22.4-cookie-parse-error-title: CookieParseError-original_id: cookie-parse-error------<a name="cookieparseerror"></a>-------<a name="exports.cookieparseerror"></a>--## `new CookieParseError(cookieHeaderString)`--**Parameters**:--- **`cookieHeaderString`**: `string`-----diff --git a/website/versioned_docs/version-0.22.4/api/GlobalCache.md b/website/versioned_docs/version-0.22.4/api/GlobalCache.mddeleted file mode 100644index ecc28faee..000000000--- a/website/versioned_docs/version-0.22.4/api/GlobalCache.md+++ /dev/null@@ -1,12 +0,0 @@-----id: version-0.22.4-global-cache-title: GlobalCache-original_id: global-cache------<a name="globalcache"></a>--Used to manage all globally created caches, such as request queue cache or dataset cache. Before creation of this class, those caches were created as-module scoped globals - untouchable. This proved problematic especially in tests, where caches would prevent test separation.-----diff --git a/website/versioned_docs/version-0.22.4/api/Job.md b/website/versioned_docs/version-0.22.4/api/Job.mddeleted file mode 100644index f0fc425a4..000000000--- a/website/versioned_docs/version-0.22.4/api/Job.md+++ /dev/null@@ -1,7 +0,0 @@-----id: version-0.22.4-job-title: Job-original_id: job------ERROR, Cannot find class.diff --git a/website/versioned_docs/version-0.22.4/api/StorageManager.md b/website/versioned_docs/version-0.22.4/api/StorageManager.mddeleted file mode 100644index 79543337c..000000000--- a/website/versioned_docs/version-0.22.4/api/StorageManager.md+++ /dev/null@@ -1,7 +0,0 @@-----id: version-0.22.4-storage-manager-title: StorageManager-original_id: storage-manager------ERROR, Cannot find class.diff --git a/website/versioned_docs/version-0.22.4/api/exports.md b/website/versioned_docs/version-0.22.4/api/exports.mddeleted file mode 100644index 348e181f6..000000000--- a/website/versioned_docs/version-0.22.4/api/exports.md+++ /dev/null@@ -1,24 +0,0 @@-----id: version-0.22.4-exports-title: exports-original_id: exports------<a name="exports"></a>--Abstract class with pre-defined method to connect to the Crawlers class by the "use" crawler method.-------<a name="module.exports"></a>--## `new module.exports(props)`--**Parameters**:--- **`props`**: `Object`- - **`pageUrl`**: `string`- - **`htmlContent`**: `string`- - **`screenshotIndex`**: `number`-----diff --git a/website/versioned_docs/version-0.22.4/typedefs/PlaywrightCrawlerOptions.md b/website/versioned_docs/version-0.22.4/typedefs/PlaywrightCrawlerOptions.mddeleted file mode 100644index d4765b710..000000000--- a/website/versioned_docs/version-0.22.4/typedefs/PlaywrightCrawlerOptions.md+++ /dev/null@@ -1,253 +0,0 @@-----id: version-0.22.4-playwright-crawler-options-title: PlaywrightCrawlerOptions-original_id: playwright-crawler-options------<a name="playwrightcrawleroptions"></a>--## Properties--### `launcher`--**Type**: `object`--Playwright launcher used for launching new browsers. For example: `require("playwright").chromium`-[Playwright list of available modules](https://playwright.dev/docs/api/class-playwright)-------### `handlePageFunction`--**Type**: `function`--Function that is called to process each request. It is passed an object with the following fields:--```-{- request: Request,- response: Response,- page: Page,- session: Session,- browserController: BrowserController,- proxyInfo: ProxyInfo,- crawler: PlaywrightCrawler,-}-```--`request` is an instance of the [`Request`](../api/request) object with details about the URL to open, HTTP method etc. `page` is an instance of the-`Playwright` [`Page`](https://playwright.dev/docs/api/class-page) `browserPool` is an instance of the-[`BrowserPool`](https://github.com/apify/browser-pool#BrowserPool), `browserController` is an instance of the-[`BrowserController`](https://github.com/apify/browser-pool#browsercontroller), `response` is an instance of the `Playwright`-[`Response`](https://playwright.dev/docs/api/class-response), which is the main resource response as returned by `page.goto(request.url)`. The-function must return a promise, which is then awaited by the crawler.--If the function throws an exception, the crawler will try to re-crawl the request later, up to `option.maxRequestRetries` times. If all the retries-fail, the crawler calls the function provided to the `handleFailedRequestFunction` parameter. To make this work, you should **always** let your-function throw exceptions rather than catch them. The exceptions are logged to the request using the-[`Request.pushErrorMessage()`](../api/request#pusherrormessage) function.-------### `navigationTimeoutSecs`--**Type**: `number` <code> = 60</code>--Timeout in which page navigation needs to finish, in seconds. When `gotoFunction()` is used and thus the default function is overridden, this timeout-will not be used and needs to be configured in the new `gotoFunction()`.-------### `handleFailedRequestFunction`--**Type**: [`HandleFailedRequest`](../typedefs/handle-failed-request)--A function to handle requests that failed more than `option.maxRequestRetries` times.--The function receives the following object as an argument:--```-{- request: Request,- response: Response,- page: Page,- session: Session,- browserController: BrowserController,- proxyInfo: ProxyInfo,- crawler: PlaywrightCrawler,-}-```--Where the [`Request`](../api/request) instance corresponds to the failed request, and the `Error` instance represents the last error thrown during-processing of the request.-------### `preNavigationHooks`--**Type**: `Array<function()>`--Async functions that are sequentially evaluated before the navigation. Good for setting additional cookies or browser properties before navigation.-The function accepts two parameters, `crawlingContext` and `gotoOptions`, which are passed to the `gotoFunction` the crawler calls to navigate.-Example:--```-preNavigationHooks: [- async (crawlingContext, gotoOptions) => {- await page.evaluate((attr) => { window.foo = attr; }, 'bar');- }-]-```-------### `postNavigationHooks`--**Type**: `Array<function()>`--Async functions that are sequentially evaluated after the navigation. Good for checking if the navigation was successful. The function accepts-`crawlingContext` as an only parameter. Example:--```-postNavigationHooks: [- async (crawlingContext) => {- const { page } = crawlingContext;- if (hasCaptcha(page)) {- await solveCaptcha (page);- }- };-]-```-------### `launchContext`--**Type**: [`PlaywrightLaunchContext`](../typedefs/playwright-launch-context)--The same options as used by [`Apify.launchPlaywright()`](../api/apify#launchplaywright).-------### `handlePageTimeoutSecs`--**Type**: `number` <code> = 60</code>--Timeout in which the function passed as `handlePageFunction` needs to finish, in seconds.-------### `browserPoolOptions`--**Type**: `BrowserPoolOptions`--Custom options passed to the underlying [`BrowserPool`](https://github.com/apify/browser-pool#BrowserPool) constructor. You can tweak those to-fine-tune browser management.-------### `persistCookiesPerSession`--**Type**: `boolean` <code> = false</code>--Automatically saves cookies to Session. Works only if Session Pool is used.-------### `proxyConfiguration`--**Type**: [`ProxyConfiguration`](../api/proxy-configuration)--If set, `PlaywrightCrawler` will be configured for all connections to use [Apify Proxy](https://my.apify.com/proxy) or your own Proxy URLs provided-and rotated according to the configuration. For more information, see the [documentation](https://docs.apify.com/proxy).-------### `requestList`--**Type**: [`RequestList`](../api/request-list)--Static list of URLs to be processed. Either `requestList` or `requestQueue` option must be provided (or both).-------### `requestQueue`--**Type**: [`RequestQueue`](../api/request-queue)--Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites. Either `requestList` or `requestQueue` option must be-provided (or both).-------### `handleRequestTimeoutSecs`--**Type**: `number` <code> = 60</code>--Timeout in which the function passed as `handleRequestFunction` needs to finish, in seconds.-------### `maxRequestRetries`--**Type**: `number` <code> = 3</code>--Indicates how many times the request is retried if-[`PuppeteerCrawlerOptions.handlePageFunction`](../typedefs/puppeteer-crawler-options#handlepagefunction) fails.-------### `maxRequestsPerCrawl`--**Type**: `number`--Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached. Always set this value in order to prevent infinite-loops in misconfigured crawlers. Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.-------### `autoscaledPoolOptions`--**Type**: [`AutoscaledPoolOptions`](../typedefs/autoscaled-pool-options)--Custom options passed to the underlying [`AutoscaledPool`](../api/autoscaled-pool) constructor. Note that the `runTaskFunction` and-`isTaskReadyFunction` options are provided by `BasicCrawler` and cannot be overridden. However, you can provide a custom implementation of-`isFinishedFunction`.-------### `minConcurrency`--**Type**: `number` <code> = 1</code>--Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding [`AutoscaledPool`](../api/autoscaled-pool) option.--_WARNING:_ If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash. If-you're not sure, just keep the default value and the concurrency will scale up automatically.-------### `maxConcurrency`--**Type**: `number` <code> = 1000</code>--Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding [`AutoscaledPool`](../api/autoscaled-pool) option.-------### `useSessionPool`--**Type**: `boolean` <code> = false</code>--If set to true. Basic crawler will initialize the [`SessionPool`](../api/session-pool) with the corresponding `sessionPoolOptions`. The session-instance will be than available in the `handleRequestFunction`.-------### `sessionPoolOptions`--**Type**: [`SessionPoolOptions`](../typedefs/session-pool-options)--The configuration options for [`SessionPool`](../api/session-pool) to use.-----diff --git a/website/versioned_docs/version-0.22.4/typedefs/PlaywrightLaunchContext.md b/website/versioned_docs/version-0.22.4/typedefs/PlaywrightLaunchContext.mddeleted file mode 100644index df5fd461a..000000000--- a/website/versioned_docs/version-0.22.4/typedefs/PlaywrightLaunchContext.md+++ /dev/null@@ -1,49 +0,0 @@-----id: version-0.22.4-playwright-launch-context-title: PlaywrightLaunchContext-original_id: playwright-launch-context------<a name="playwrightlaunchcontext"></a>--Apify extends the launch options of Playwright. You can use any of the Playwright compatible-[`LaunchOptions`](https://playwright.dev/docs/api/class-browsertype#browsertypelaunchoptions) options by providing the `launchOptions` property.--## Properties--### `launchOptions`--**Type**: `object`--`browserType.launch` [options](https://playwright.dev/docs/api/class-browsertype?_highlight=launch#browsertypelaunchoptions)-------### `proxyUrl`--**Type**: `string`--URL to a HTTP proxy server. It must define the port number, and it may also contain proxy username and password.--Example: `http://bob:[email protected]:1234`.-------### `useChrome`--**Type**: `boolean` <code> = false</code>--If `true` and `executablePath` is not set, Puppeteer will launch full Google Chrome browser available on the machine rather than the bundled Chromium.-The path to Chrome executable is taken from the `APIFY_CHROME_EXECUTABLE_PATH` environment variable if provided, or defaults to the typical Google-Chrome executable location specific for the operating system. By default, this option is `false`.-------### `launcher`--**Type**: `string` | `Object`--By default this function uses-require("playwright").chromium`. If you want to use a different browser you can pass it by this property as`require("playwright").firefox-----