Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

23.2 KB
Raw Blame

Change Log

All notable changes to this project will be documented in this file. See Conventional Commits for commit guidelines.

3.18.1 (2026-08-12)

Note: Version bump only for package @crawlee/utils

3.18.0 (2026-08-04)

Bug Fixes

  • filter sitemap-derived URLs by enqueue strategy (#3797) (9dc3160)
  • sitemap: recognize sitemap_index as a valid input sitemap url (#3909) (eacab5e)
  • utils: advance the CPU tick baseline in systemInfoV2 getCurrentCpuTicks (#3824) (bb23800)
  • utils: allow apostrophes and asterisks in extracted URLs (#3790) (e9baffd), closes #2755
  • utils: prevent ReDoS in social email regex (#3845) (d6a2b39)
  • utils: prevent sitemap metadata leak across url without loc (#3795) (895ed62)
  • utils: use article:* selectors for article Open Graph properties (#3789) (b660506)

3.17.0 (2026-06-04)

Bug Fixes

  • add network timeouts to discoverValidSitemaps to prevent indefinite hangs (#3429) (ee90835)
  • core: apply object searchParams in HTTP request options (#3622) (d32946e)
  • log warning when Sitemap.load returns empty sitemap due to loading error (#3462) (2cf9c1c)
  • Prevent accidental request dropping with maxRequestsPerCrawl (#3531) (b23319b)
  • prevent v3 sitemap discovery init hangs in discoverValidSitemaps (#3434) (6c04f92), closes #3412
  • using GET for candidate checks - sitemaps (#3464) (f82eb04), closes #3463

Features

  • utils: add sitemapFilter option to parseSitemap (#3557) (1d4f6b9)

3.16.0 (2026-02-06)

Bug Fixes

Features

  • utils: add discoverValidSitemaps utility (#3339) (29f52ed)

3.15.3 (2025-11-10)

Note: Version bump only for package @crawlee/utils

3.15.2 (2025-10-23)

Features

  • export cheerio types in all crawler packages (#3204) (f05790b)

3.15.1 (2025-09-26)

Note: Version bump only for package @crawlee/utils

3.15.0 (2025-09-17)

Note: Version bump only for package @crawlee/utils

3.14.1 (2025-08-05)

Note: Version bump only for package @crawlee/utils

3.14.0 (2025-07-25)

Bug Fixes

  • validation of iterables when adding requests to the queue (#3091) (529a1dd), closes #3063

3.13.10 (2025-07-09)

Note: Version bump only for package @crawlee/utils

3.13.9 (2025-06-27)

Bug Fixes

  • Do not log 'malformed sitemap content' on network errors in Sitemap.tryCommonNames (#3015) (64a090f), closes #2884

Features

3.13.8 (2025-06-16)

Bug Fixes

  • Persist rendering type detection results in AdaptivePlaywrightCrawler (#2987) (76431ba), closes #2899

3.13.7 (2025-06-06)

Note: Version bump only for package @crawlee/utils

3.13.6 (2025-06-05)

Note: Version bump only for package @crawlee/utils

3.13.5 (2025-05-20)

Note: Version bump only for package @crawlee/utils

3.13.4 (2025-05-14)

Bug Fixes

  • social: extract emails from each text node separately (#2952) (799afc1)

3.13.3 (2025-05-05)

Note: Version bump only for package @crawlee/utils

3.13.2 (2025-04-08)

Note: Version bump only for package @crawlee/utils

3.13.1 (2025-04-07)

Bug Fixes

Features

3.13.0 (2025-03-04)

Features

3.12.2 (2025-01-27)

Note: Version bump only for package @crawlee/utils

3.12.1 (2024-12-04)

Bug Fixes

  • social: support new URL formats for Facebook, YouTube and X (#2758) (4c95847), closes #525

3.12.0 (2024-11-04)

Bug Fixes

Features

3.11.5 (2024-10-04)

Note: Version bump only for package @crawlee/utils

3.11.4 (2024-09-23)

Bug Fixes

3.11.3 (2024-09-03)

Bug Fixes

  • improve FACEBOOK_REGEX to match older style page URLs (#2650) (a005e69), closes #2216

3.11.2 (2024-08-28)

Bug Fixes

  • use namespace imports for cheerio to be compatible with v1 (#2641) (f48296f)

Features

3.11.1 (2024-07-24)

Bug Fixes

3.11.0 (2024-07-09)

Features

  • Sitemap-based request list implementation (#2498) (7bf8f0b)

3.10.5 (2024-06-12)

Note: Version bump only for package @crawlee/utils

3.10.4 (2024-06-11)

Note: Version bump only for package @crawlee/utils

3.10.3 (2024-06-07)

Bug Fixes

  • respect implicit router when no requestHandler is provided in AdaptiveCrawler (#2518) (31083aa)

3.10.2 (2024-06-03)

Bug Fixes

Features

3.10.1 (2024-05-23)

Bug Fixes

  • adjust URL_NO_COMMAS_REGEX regexp to allow single character hostnames (#2492) (ec802e8), closes #2487

3.10.0 (2024-05-16)

Bug Fixes

  • malformed sitemap url when sitemap index child contains querystring (#2430) (e4cd41c)
  • return true when robots.isAllowed returns undefined (#2439) (6f541f8), closes #2437
  • sitemap content-type check breaks on content-type parameters (#2442) (db7d372)

Features

  • implement ErrorSnapshotter for error context capture (#2332) (e861dfd), closes #2280

3.9.2 (2024-04-17)

Features

3.9.1 (2024-04-11)

Note: Version bump only for package @crawlee/utils

3.9.0 (2024-04-10)

Bug Fixes

Features

  • expand #shadow-root elements automatically in parseWithCheerio helper (#2396) (a05b3a9)

3.8.2 (2024-03-21)

Bug Fixes

  • correctly report gzip decompression errors (#2368) (84a2f17)

3.8.1 (2024-02-22)

Note: Version bump only for package @crawlee/utils

3.8.0 (2024-02-21)

Features

  • add Sitemap.tryCommonNames to check well known sitemap locations (#2311) (85589f1), closes #2307
  • core: add userAgent parameter to RobotsFile.isAllowed() + RobotsFile.from() helper (#2338) (343c159)
  • Support plain-text sitemap files (sitemap.txt) (#2315) (0bee7da)

3.7.3 (2024-01-30)

Bug Fixes

3.7.2 (2024-01-09)

Note: Version bump only for package @crawlee/utils

3.7.1 (2024-01-02)

Bug Fixes

  • ES2022 build compatibility and move to NodeNext for module (#2258) (7fe1e68), closes #2257

3.7.0 (2023-12-21)

Bug Fixes

  • retryOnBlocked doesn't override the blocked HTTP codes (#2243) (81672c3)

Features

3.6.2 (2023-11-26)

Note: Version bump only for package @crawlee/utils

3.6.1 (2023-11-15)

Note: Version bump only for package @crawlee/utils

3.6.0 (2023-11-15)

Features

3.5.8 (2023-10-17)

Bug Fixes

  • refactor extractUrls to split the text line by line first (#2122) (7265cd7)

3.5.7 (2023-10-05)

Note: Version bump only for package @crawlee/utils

3.5.6 (2023-10-04)

Features

3.5.5 (2023-10-02)

Note: Version bump only for package @crawlee/utils

3.5.4 (2023-09-11)

Note: Version bump only for package @crawlee/utils

3.5.3 (2023-08-31)

Bug Fixes

3.5.2 (2023-08-21)

Note: Version bump only for package @crawlee/utils

3.5.1 (2023-08-16)

Note: Version bump only for package @crawlee/utils

3.5.0 (2023-07-31)

Features

3.4.2 (2023-07-19)

Features

3.4.1 (2023-07-13)

Note: Version bump only for package @crawlee/utils

3.4.0 (2023-06-12)

Note: Version bump only for package @crawlee/utils

3.3.3 (2023-05-31)

Note: Version bump only for package @crawlee/utils

3.3.2 (2023-05-11)

Note: Version bump only for package @crawlee/utils

3.3.1 (2023-04-11)

Bug Fixes

  • jsdom: delay closing of the window and add some polyfills (2e81618)

3.3.0 (2023-03-09)

Bug Fixes

  • add proxyUrl to DownloadListOfUrlsOptions (779be1e), closes #1780

3.2.2 (2023-02-08)

Note: Version bump only for package @crawlee/utils

3.2.1 (2023-02-07)

Note: Version bump only for package @crawlee/utils

3.2.0 (2023-02-07)

Bug Fixes

  • utils: add missing dependency on ow (bf0e03c), closes #1716

3.1.2 (2022-11-15)

Note: Version bump only for package @crawlee/utils

3.1.1 (2022-11-07)

Note: Version bump only for package @crawlee/utils

3.1.0 (2022-10-13)

Note: Version bump only for package @crawlee/utils

3.0.4 (2022-08-22)

Note: Version bump only for package @crawlee/utils