Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
docs: document slim Docker images and package manager setup (#4082)
Vlad Frangube4a0bf2d133
diff --git a/docs/guides/docker_browser_js.txt b/docs/guides/docker_browser_js.txtindex 76848bac3..215a99e07 100644--- a/docs/guides/docker_browser_js.txt+++ b/docs/guides/docker_browser_js.txt@@ -1,7 +1,7 @@# Specify the base Docker image. You can read more about# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.-FROM apify/actor-node-playwright-chrome:20+FROM apify/actor-node-playwright-chrome:24# Copy just package.json and package-lock.json# to speed up the build using Docker layer cache.diff --git a/docs/guides/docker_browser_ts.txt b/docs/guides/docker_browser_ts.txtindex 513bccfc1..63567b47a 100644--- a/docs/guides/docker_browser_ts.txt+++ b/docs/guides/docker_browser_ts.txt@@ -1,7 +1,7 @@# Specify the base Docker image. You can read more about# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.-FROM apify/actor-node-playwright-chrome:20 AS builder+FROM apify/actor-node-playwright-chrome:24 AS builder# Copy just package.json and package-lock.json# to speed up the build using Docker layer cache.@@ -19,7 +19,7 @@ COPY --chown=myuser . ./RUN npm run build# Create final image-FROM apify/actor-node-playwright-chrome:20+FROM apify/actor-node-playwright-chrome:24# Copy only built JS files from builder imageCOPY --from=builder --chown=myuser /home/myuser/dist ./distdiff --git a/docs/guides/docker_images.mdx b/docs/guides/docker_images.mdxindex 1129fc804..6a909d140 100644--- a/docs/guides/docker_images.mdx+++ b/docs/guides/docker_images.mdx@@ -29,6 +29,23 @@ Browsers are pretty big, so we try to provide a wide variety of images to suit t- [`apify/actor-node-playwright-chrome`](#actor-node-playwright-chrome)- [`apify/actor-node-playwright-firefox`](#actor-node-playwright-firefox)- [`apify/actor-node-playwright-webkit`](#actor-node-playwright-webkit)+- [`apify/actor-node-playwright-camoufox`](#actor-node-playwright-camoufox)++Every image is published in two flavours. The full image preinstalls `apify`, `crawlee` and `typescript`. The `-slim` variant (e.g. `24-slim`, `24-1.60.0-slim`, `24-beta-slim`) only ships the browser automation library the image is built around (for example `puppeteer`, `playwright`, or `camoufox-js` with `impit`), and `apify/actor-node:24-slim` ships no npm packages at all. Slim images are smaller and faster to pull, and your `package.json` is the single source of truth for dependency versions.++Use the slim variant unless you have a reason not to. Reach for the full image when you want to run something quickly without maintaining a `package.json`, or when you rely on the exact preinstalled versions of `apify` and `crawlee`.++```dockerfile+# No preinstalled packages, bring your own dependencies.+FROM apify/actor-node:24-slim+```++```dockerfile+# Only Playwright (Chromium) is preinstalled, pinned to match the bundled browser.+FROM apify/actor-node-playwright-chrome:24-1.60.0-slim+```++Since nothing but the automation library is preinstalled, make sure `crawlee` and any other runtime dependencies are listed in your `package.json` and installed in your `Dockerfile`. The [version matching](#recommended-approach-pin-both-versions) rules for the automation library still apply.## Versioning@@ -38,7 +55,7 @@ Each image is tagged with up to 2 version tags, depending on the type of the ima### Node.js versioning-Our images are built with multiple Node.js versions to ensure backwards compatibility. Currently, Node.js **versions 20, 22, and 24 are supported** (legacy versions still exist, see DockerHub). To select the preferred version, use the appropriate number as the image tag.+Our images are built with multiple Node.js versions to ensure backwards compatibility. Currently, Node.js **versions 22, 24, and 26 are supported** (legacy versions still exist, see DockerHub). To select the preferred version, use the appropriate number as the image tag.```dockerfile# Use Node.js 24@@ -67,6 +84,35 @@ FROM apify/actor-node:24-betaFROM apify/actor-node-playwright-chrome:24-1.60.0-beta```+## Node.js package managers++All Node.js images ship with npm and have [Corepack](https://github.com/nodejs/corepack) enabled, so you can use yarn or pnpm as well. Neither is preinstalled: add a [`packageManager`](https://nodejs.org/api/packages.html#packagemanager) field to your `package.json` and Corepack downloads and uses the exact version you pin.++```json+{+ "packageManager": "[email protected]"+}+```++The images preconfigure the package managers so that:++- pnpm and yarn install a flat, npm-style `node_modules` (`node-linker=hoisted` for pnpm, `nodeLinker: node-modules` for yarn) instead of a symlinked store or Plug'n'Play, so dependencies resolve without extra loaders.+- The yarn and pnpm caches (`YARN_CACHE_FOLDER`, `YARN_GLOBAL_FOLDER`, `PNPM_CONFIG_STORE_DIR`, `PNPM_CONFIG_CACHE_DIR`) and the Corepack cache (`COREPACK_HOME`) live under `/pkg-cache`. npm keeps its default `~/.npm` cache. Both directories only hold throwaway data, so you can `rm -rf /pkg-cache/* ~/.npm` at the end of your `Dockerfile` to reclaim space without touching installed dependencies.++:::note Overriding the linker++The images set the linker through the `PNPM_CONFIG_NODE_LINKER` and `YARN_NODE_LINKER` environment variables. Both pnpm and yarn give environment variables precedence over `.npmrc` or `.yarnrc.yml`, so a config file alone does not change the linker. To use a different one, override the variable in your `Dockerfile`:++```dockerfile+# https://pnpm.io/settings#nodelinker+ENV PNPM_CONFIG_NODE_LINKER=isolated+# https://yarnpkg.com/configuration/yarnrc#nodeLinker+ENV YARN_NODE_LINKER=pnp+```++:::++## Best practicesFor production crawlers, we recommend pinning both the Node.js version **and** the automation library version in your Dockerfile tag. This ensures reproducible builds and prevents unexpected behavior when new versions are released.@@ -132,7 +178,7 @@ curl -s "https://registry.hub.docker.com/v2/repositories/apify/actor-node-playwr### Warning about image size-Browsers are huge. If you don't need them all in your image, it's better to use a smaller image with only the one browser you need.+Browsers are huge. If you don't need them all in your image, it's better to use a smaller image with only the one browser you need. If you don't need the preinstalled `apify` and `crawlee` packages either, use the [`-slim` variant](#overview).You should also be careful when installing new dependencies. Nothing prevents you from installing Playwright into the`actor-node-puppeteer-chrome` image, but the resulting image will be about 3 times larger and extremely slow to download and build.@@ -198,6 +244,14 @@ pre-installed.FROM apify/actor-node-playwright-webkit:24```+### actor-node-playwright-camoufox++Same idea as [`actor-node-playwright-firefox`](#actor-node-playwright-firefox), but with [Camoufox](https://camoufox.com/), a Firefox fork hardened against bot detection, pre-installed instead of Firefox. The image also ships the [`camoufox-js`](https://github.com/apify/camoufox-js) and [`impit`](https://github.com/apify/impit) packages.++```dockerfile+FROM apify/actor-node-playwright-camoufox:24+```+## Example DockerfileTo use the above images, it's necessary to have a [`Dockerfile`](https://docs.docker.com/engine/reference/builder/). You can either use this example, or bootstrap your projects with the [Crawlee CLI](../introduction/setting-up) which automatically adds the correct Dockerfile into our project folder.diff --git a/docs/guides/docker_node_js.txt b/docs/guides/docker_node_js.txtindex 50cecc043..a86d9763c 100644--- a/docs/guides/docker_node_js.txt+++ b/docs/guides/docker_node_js.txt@@ -1,7 +1,7 @@# Specify the base Docker image. You can read more about# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.-FROM apify/actor-node:20+FROM apify/actor-node:24# Copy just package.json and package-lock.json# to speed up the build using Docker layer cache.diff --git a/docs/guides/docker_node_ts.txt b/docs/guides/docker_node_ts.txtindex 30d85ce51..20812812c 100644--- a/docs/guides/docker_node_ts.txt+++ b/docs/guides/docker_node_ts.txt@@ -1,7 +1,7 @@# Specify the base Docker image. You can read more about# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.-FROM apify/actor-node:20 AS builder+FROM apify/actor-node:24 AS builder# Copy just package.json and package-lock.json# to speed up the build using Docker layer cache.@@ -19,7 +19,7 @@ COPY . ./RUN npm run build# Create final image-FROM apify/actor-node:20+FROM apify/actor-node:24# Copy only built JS files from builder imageCOPY --from=builder /usr/src/app/dist ./distdiff --git a/website/versioned_docs/version-3.18/guides/docker_browser_js.txt b/website/versioned_docs/version-3.18/guides/docker_browser_js.txtindex 76848bac3..215a99e07 100644--- a/website/versioned_docs/version-3.18/guides/docker_browser_js.txt+++ b/website/versioned_docs/version-3.18/guides/docker_browser_js.txt@@ -1,7 +1,7 @@# Specify the base Docker image. You can read more about# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.-FROM apify/actor-node-playwright-chrome:20+FROM apify/actor-node-playwright-chrome:24# Copy just package.json and package-lock.json# to speed up the build using Docker layer cache.diff --git a/website/versioned_docs/version-3.18/guides/docker_browser_ts.txt b/website/versioned_docs/version-3.18/guides/docker_browser_ts.txtindex 513bccfc1..63567b47a 100644--- a/website/versioned_docs/version-3.18/guides/docker_browser_ts.txt+++ b/website/versioned_docs/version-3.18/guides/docker_browser_ts.txt@@ -1,7 +1,7 @@# Specify the base Docker image. You can read more about# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.-FROM apify/actor-node-playwright-chrome:20 AS builder+FROM apify/actor-node-playwright-chrome:24 AS builder# Copy just package.json and package-lock.json# to speed up the build using Docker layer cache.@@ -19,7 +19,7 @@ COPY --chown=myuser . ./RUN npm run build# Create final image-FROM apify/actor-node-playwright-chrome:20+FROM apify/actor-node-playwright-chrome:24# Copy only built JS files from builder imageCOPY --from=builder --chown=myuser /home/myuser/dist ./distdiff --git a/website/versioned_docs/version-3.18/guides/docker_images.mdx b/website/versioned_docs/version-3.18/guides/docker_images.mdxindex 1129fc804..6a909d140 100644--- a/website/versioned_docs/version-3.18/guides/docker_images.mdx+++ b/website/versioned_docs/version-3.18/guides/docker_images.mdx@@ -29,6 +29,23 @@ Browsers are pretty big, so we try to provide a wide variety of images to suit t- [`apify/actor-node-playwright-chrome`](#actor-node-playwright-chrome)- [`apify/actor-node-playwright-firefox`](#actor-node-playwright-firefox)- [`apify/actor-node-playwright-webkit`](#actor-node-playwright-webkit)+- [`apify/actor-node-playwright-camoufox`](#actor-node-playwright-camoufox)++Every image is published in two flavours. The full image preinstalls `apify`, `crawlee` and `typescript`. The `-slim` variant (e.g. `24-slim`, `24-1.60.0-slim`, `24-beta-slim`) only ships the browser automation library the image is built around (for example `puppeteer`, `playwright`, or `camoufox-js` with `impit`), and `apify/actor-node:24-slim` ships no npm packages at all. Slim images are smaller and faster to pull, and your `package.json` is the single source of truth for dependency versions.++Use the slim variant unless you have a reason not to. Reach for the full image when you want to run something quickly without maintaining a `package.json`, or when you rely on the exact preinstalled versions of `apify` and `crawlee`.++```dockerfile+# No preinstalled packages, bring your own dependencies.+FROM apify/actor-node:24-slim+```++```dockerfile+# Only Playwright (Chromium) is preinstalled, pinned to match the bundled browser.+FROM apify/actor-node-playwright-chrome:24-1.60.0-slim+```++Since nothing but the automation library is preinstalled, make sure `crawlee` and any other runtime dependencies are listed in your `package.json` and installed in your `Dockerfile`. The [version matching](#recommended-approach-pin-both-versions) rules for the automation library still apply.## Versioning@@ -38,7 +55,7 @@ Each image is tagged with up to 2 version tags, depending on the type of the ima### Node.js versioning-Our images are built with multiple Node.js versions to ensure backwards compatibility. Currently, Node.js **versions 20, 22, and 24 are supported** (legacy versions still exist, see DockerHub). To select the preferred version, use the appropriate number as the image tag.+Our images are built with multiple Node.js versions to ensure backwards compatibility. Currently, Node.js **versions 22, 24, and 26 are supported** (legacy versions still exist, see DockerHub). To select the preferred version, use the appropriate number as the image tag.```dockerfile# Use Node.js 24@@ -67,6 +84,35 @@ FROM apify/actor-node:24-betaFROM apify/actor-node-playwright-chrome:24-1.60.0-beta```+## Node.js package managers++All Node.js images ship with npm and have [Corepack](https://github.com/nodejs/corepack) enabled, so you can use yarn or pnpm as well. Neither is preinstalled: add a [`packageManager`](https://nodejs.org/api/packages.html#packagemanager) field to your `package.json` and Corepack downloads and uses the exact version you pin.++```json+{+ "packageManager": "[email protected]"+}+```++The images preconfigure the package managers so that:++- pnpm and yarn install a flat, npm-style `node_modules` (`node-linker=hoisted` for pnpm, `nodeLinker: node-modules` for yarn) instead of a symlinked store or Plug'n'Play, so dependencies resolve without extra loaders.+- The yarn and pnpm caches (`YARN_CACHE_FOLDER`, `YARN_GLOBAL_FOLDER`, `PNPM_CONFIG_STORE_DIR`, `PNPM_CONFIG_CACHE_DIR`) and the Corepack cache (`COREPACK_HOME`) live under `/pkg-cache`. npm keeps its default `~/.npm` cache. Both directories only hold throwaway data, so you can `rm -rf /pkg-cache/* ~/.npm` at the end of your `Dockerfile` to reclaim space without touching installed dependencies.++:::note Overriding the linker++The images set the linker through the `PNPM_CONFIG_NODE_LINKER` and `YARN_NODE_LINKER` environment variables. Both pnpm and yarn give environment variables precedence over `.npmrc` or `.yarnrc.yml`, so a config file alone does not change the linker. To use a different one, override the variable in your `Dockerfile`:++```dockerfile+# https://pnpm.io/settings#nodelinker+ENV PNPM_CONFIG_NODE_LINKER=isolated+# https://yarnpkg.com/configuration/yarnrc#nodeLinker+ENV YARN_NODE_LINKER=pnp+```++:::++## Best practicesFor production crawlers, we recommend pinning both the Node.js version **and** the automation library version in your Dockerfile tag. This ensures reproducible builds and prevents unexpected behavior when new versions are released.@@ -132,7 +178,7 @@ curl -s "https://registry.hub.docker.com/v2/repositories/apify/actor-node-playwr### Warning about image size-Browsers are huge. If you don't need them all in your image, it's better to use a smaller image with only the one browser you need.+Browsers are huge. If you don't need them all in your image, it's better to use a smaller image with only the one browser you need. If you don't need the preinstalled `apify` and `crawlee` packages either, use the [`-slim` variant](#overview).You should also be careful when installing new dependencies. Nothing prevents you from installing Playwright into the`actor-node-puppeteer-chrome` image, but the resulting image will be about 3 times larger and extremely slow to download and build.@@ -198,6 +244,14 @@ pre-installed.FROM apify/actor-node-playwright-webkit:24```+### actor-node-playwright-camoufox++Same idea as [`actor-node-playwright-firefox`](#actor-node-playwright-firefox), but with [Camoufox](https://camoufox.com/), a Firefox fork hardened against bot detection, pre-installed instead of Firefox. The image also ships the [`camoufox-js`](https://github.com/apify/camoufox-js) and [`impit`](https://github.com/apify/impit) packages.++```dockerfile+FROM apify/actor-node-playwright-camoufox:24+```+## Example DockerfileTo use the above images, it's necessary to have a [`Dockerfile`](https://docs.docker.com/engine/reference/builder/). You can either use this example, or bootstrap your projects with the [Crawlee CLI](../introduction/setting-up) which automatically adds the correct Dockerfile into our project folder.diff --git a/website/versioned_docs/version-3.18/guides/docker_node_js.txt b/website/versioned_docs/version-3.18/guides/docker_node_js.txtindex 50cecc043..a86d9763c 100644--- a/website/versioned_docs/version-3.18/guides/docker_node_js.txt+++ b/website/versioned_docs/version-3.18/guides/docker_node_js.txt@@ -1,7 +1,7 @@# Specify the base Docker image. You can read more about# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.-FROM apify/actor-node:20+FROM apify/actor-node:24# Copy just package.json and package-lock.json# to speed up the build using Docker layer cache.diff --git a/website/versioned_docs/version-3.18/guides/docker_node_ts.txt b/website/versioned_docs/version-3.18/guides/docker_node_ts.txtindex 30d85ce51..20812812c 100644--- a/website/versioned_docs/version-3.18/guides/docker_node_ts.txt+++ b/website/versioned_docs/version-3.18/guides/docker_node_ts.txt@@ -1,7 +1,7 @@# Specify the base Docker image. You can read more about# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.-FROM apify/actor-node:20 AS builder+FROM apify/actor-node:24 AS builder# Copy just package.json and package-lock.json# to speed up the build using Docker layer cache.@@ -19,7 +19,7 @@ COPY . ./RUN npm run build# Create final image-FROM apify/actor-node:20+FROM apify/actor-node:24# Copy only built JS files from builder imageCOPY --from=builder /usr/src/app/dist ./dist