Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
How Crawlee releases work
Releases are managed by GitHub Actions and lerna. There are two release lines:
| line | branch | stable dist-tag | canary dist-tag |
|---|---|---|---|
| v4 (current development) | master |
latest (from 4.0.0 on) |
v4 now, next once 4.0.0 is stable |
| v3 (maintenance) | 3.x |
latest until 4.0.0 ships, then latest-v3 |
next now, next-v3 once 4.0.0 is stable |
Release candidates for the next major are published from master under the rc dist-tag.
Canary releases
Every push to master or 3.x (that is not a docs change and does not say [skip ci])
triggers the release_next job in test-ci.yml. It builds the packages, derives a canary
version from lerna.json (scripts/copy.ts --canary) and invokes publish-to-npm.yml,
which publishes all packages under the branch's canary dist-tag (see the table above; the
tag is hardcoded in the branch's publish:next script in package.json). No commit or git
tag is created for canaries.
Stable releases
Trigger the Release @latest workflow (release.yml) manually via workflow_dispatch
from the branch you want to release:
master+bump: majoris how4.0.0goes out (lerna bumps3.18.x→4.0.0, publishes tolatest).3.x+bump: patch/minorships v3 maintenance releases.
The workflow runs the tests, bumps versions via lerna version (conventional-commits
changelog, GitHub release, git tag), pins internal dependency versions, publishes through
publish-to-npm.yml with dist-tag: prod, and triggers Apify Docker image builds.
On minor/major releases from master, the version-docs job snapshots the current docs
into website/versioned_docs (Docusaurus docs:version + api:version). The job never
runs for maintenance branches — 3.x docs live in the version-3.18 snapshot on master.
RC releases
Dispatch publish-to-npm.yml from master with dist-tag: rc. This publishes
4.0.0-rc.N under the rc dist-tag and pushes a v4.0.0-rc.N git tag pointing at the
released commit; no version-bump commit lands on the branch.
v4.0.0 release day
See the tracking issue for the ordered checklist (publish 4.0.0 to latest, switch
master canaries v4 → next and --canary=major → --canary=patch, switch 3.x to
latest-v3/next-v3, drop the RC docs label).
Playbook: switching master to the next major
Notes from the v4 transition (August 2026), for whoever does v5. The overall flow: develop
the next major on a vN branch (canaries under the vN dist-tag, RCs under rc), and when
it is time to make it the main line, cut a maintenance branch and fast-forward master.
- Cut the maintenance branch first. Branch
(N-1).xoff the master tip. In one commit, adapt its workflows:test-ci.ymltriggers and therelease_nextgate point at the branch (use exactgithub.ref == 'refs/heads/(N-1).x'matching, notcontains()— the substring match is a footgun), and delete theversion-docsjob fromrelease.yml(it checks out the repo default branch, so on a maintenance branch it would snapshot the wrong docs). Canaries keep thenexttag and stable releases keeplatestuntil the new major actually ships. - Rebase the
vNbranch onto the master tip and validate: no conflict markers, tree diff vs the oldvNtip only shows what you expect, full build,tsc-check-tests,api:check, test suite, website build. Watch for master-only features silently lost in the rebase — compare public API snapshots (docs/public-api/), and re-check any file both lines touched heavily. - Docs. Delete the dev snapshot (
website/versioned_docs/version-N.0, its sidebars file, and theversions.jsonentry) — the dev docs become the unversioned "current" version, labeled viaversions.current.label('N.0 (RC)') indocusaurus.config.js. The previous major stays the default docs version until the stable release, whoseversion-docsjob creates the realN.0snapshot. Gotchas: the latest snapshot is served unversioned at/js/api, so hardcoded/js/api/N-1.x/...links (blog posts, snapshot edits from the dev branch) break once the dev snapshot is gone, and package READMEs that link to new-major API pages need/js/api/next/...until the major is the default.ApiLink.jsxandNavbarItem/ComponentTypes.jsmust both derive the stable version fromversions.json[0]. - Fast-forward push master. A PR cannot do this: the org ruleset only allows squash
merges into master, and a squash (or GitHub's rebase-merge) would rewrite the history.
Direct pushes are blocked by three rulesets — org "Important branches PR enforcement",
org "Allow only squash merges…" (its
pull_requestrule also enforces PR-only pushes), and repo "Require CI checks on master" — all of which accept theBypassTemporaryteam, so join it for the duration of the push (the last two needed the team added in the v4 round; check they still have it). - Retarget the open
vN-based PRs to master yourself (gh pr edit --base master) before deleting thevNbranch. GitHub only auto-retargets when the base branch is deleted after being merged through a PR; after a fast-forward push, deleting the branch would auto-close them instead. - Update the required status checks on the "Require CI checks on master" ruleset if the
Node matrix changed, and check Renovate/Dependabot: the dev branch's lockfile format wins
after the merge, and dependency bumps that renovate landed on old-master meanwhile may
need re-applying (a stale peer-keyed lockfile entry can surface as bogus type errors —
pnpm dedupefixes that). - Open the release-day tracking issue (see the v4.0.0 section above for the shape) and
keep the dist-tag flips in it — the maintenance branch must move off
latestthe same day the new major claims it.