Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

32.3 KB
Raw Blame

Change Log

All notable changes to this project will be documented in this file. See Conventional Commits for commit guidelines.

3.18.1 (2026-08-12)

Note: Version bump only for package @crawlee/basic

3.18.0 (2026-08-04)

Bug Fixes

  • avoid duplicate final crawler persistence (#3866) (79dd6c4), closes #3167
  • basic-crawler: declare @apify/datastructures as a dependency (#3815) (4e9d227)
  • core: keep crawler enqueue limits when options carry explicit undefined (#3927) (5f1b0ab), closes #3924 #3924
  • make crawler terminal status message reliably delivered (#3733) (7aed264)
  • respect falsy transformRequestFunction return in context enqueueLinks (#3925) (6cb4fc3), closes #3920

Features

3.17.0 (2026-06-04)

Bug Fixes

  • correct failed requests delta count in periodic log message (#3482) (a098f42)
  • Correctly track the number of requests handled by a crawler (#3410) (b2296ce), closes #3083 #3330
  • honor custom respectRobotsTxtFile userAgent in enqueueLinks (#3578) (1d2528b)
  • Prevent accidental request dropping with maxRequestsPerCrawl (#3531) (b23319b)

3.16.0 (2026-02-06)

Bug Fixes

  • core: ensure maxCrawlDepth warning is logged only once (#3337) (9d01334), closes #3336
  • handle multiple BasicCrawler.stop() calls correctly (#3324) (9c0580b), closes #3257
  • suppress info message for undefined maxRequestsPerCrawl (#3237) (f3d9a79)

Performance Improvements

3.15.3 (2025-11-10)

Bug Fixes

Features

3.15.2 (2025-10-23)

Features

  • export cheerio types in all crawler packages (#3204) (f05790b)

Performance Improvements

3.15.1 (2025-09-26)

Bug Fixes

  • use correct config for storage classes to avoid memory leaks (#3144) (911a2eb)

3.15.0 (2025-09-17)

Bug Fixes

  • don't fail exportData calls on empty datasets (#3115) (298f170), closes #2734
  • respect maxCrawlDepth with a custom enqueueLinks transformRequestFunction (#3159) (e2ecb74)

Features

  • add collectAllKeys option for BasicCrawler.exportData (#3129) (2ddfc9c), closes #3007
  • add TandemRequestProvider for combined RequestList and RequestQueue usage (#2914) (4ca450f), closes #2499

3.14.1 (2025-08-05)

Note: Version bump only for package @crawlee/basic

3.14.0 (2025-07-25)

Bug Fixes

  • validation of iterables when adding requests to the queue (#3091) (529a1dd), closes #3063

Features

3.13.10 (2025-07-09)

Note: Version bump only for package @crawlee/basic

3.13.9 (2025-06-27)

Features

  • Accept (Async)Iterables in addRequests methods (#3013) (a4ab748), closes #2980
  • Report links skipped because of various filter conditions (#3026) (5a867bc), closes #3016

3.13.8 (2025-06-16)

Bug Fixes

  • Do not enqueue more links than what the crawler is capable of processing (#2990) (ea094c8), closes #2728

3.13.7 (2025-06-06)

Note: Version bump only for package @crawlee/basic

3.13.6 (2025-06-05)

Note: Version bump only for package @crawlee/basic

3.13.5 (2025-05-20)

Note: Version bump only for package @crawlee/basic

3.13.4 (2025-05-14)

Bug Fixes

  • Optimize request unlocking to get rid of unnecessary unlock calls (#2963) (a433037)

3.13.3 (2025-05-05)

Bug Fixes

  • respect autoscaledPoolOptions.isTaskReadyFunction option (#2948) (fe2d206), closes #2922
  • statistics: track actual request.retryCount in Statistics (#2940) (c9f7f54)

3.13.2 (2025-04-08)

Features

3.13.1 (2025-04-07)

Bug Fixes

Features

3.13.0 (2025-03-04)

Bug Fixes

3.12.2 (2025-01-27)

Bug Fixes

  • destructure CrawlerRunOptions before passing them to addRequests (#2803) (02a598c), closes #2802
  • graceful BasicCrawler tidy-up on CriticalError (#2817) (53331e8), closes #2807

Features

  • stopping the crawlers gracefully with BasicCrawler.stop() (#2792) (af2966f), closes #2777

3.12.1 (2024-12-04)

Bug Fixes

  • log status message timeouts to debug level (55ee44a)

3.12.0 (2024-11-04)

Features

3.11.5 (2024-10-04)

Bug Fixes

3.11.4 (2024-09-23)

Note: Version bump only for package @crawlee/basic

3.11.3 (2024-09-03)

Note: Version bump only for package @crawlee/basic

3.11.2 (2024-08-28)

Bug Fixes

  • RequestQueueV2: remove inProgress cache, rely solely on locked states (#2601) (57fcb08)

3.11.1 (2024-07-24)

Note: Version bump only for package @crawlee/basic

3.11.0 (2024-07-09)

Features

  • Sitemap-based request list implementation (#2498) (7bf8f0b)

3.10.5 (2024-06-12)

Bug Fixes

  • mark context.request.loadedUrl and id as required inside the request handler (#2531) (2b54660)

3.10.4 (2024-06-11)

Bug Fixes

  • add missing useState implementation into crawling context (eec4a71)
  • make crawler.log publicly accessible (#2526) (3e9e665)
  • respect crawler.log when creating child logger for Statistics (0a0d75d), closes #2412

3.10.3 (2024-06-07)

Features

  • log desired concurrency in the default status message (9f0b796)

3.10.2 (2024-06-03)

Note: Version bump only for package @crawlee/basic

3.10.1 (2024-05-23)

Note: Version bump only for package @crawlee/basic

3.10.0 (2024-05-16)

Bug Fixes

  • EnqueueStrategy.All erroring with links using unsupported protocols (#2389) (8db3908)
  • do not drop statistics on migration/resurrection/resume (#2462) (8ce7dd4)

Features

3.9.2 (2024-04-17)

Bug Fixes

3.9.1 (2024-04-11)

Note: Version bump only for package @crawlee/basic

3.9.0 (2024-04-10)

Bug Fixes

  • notify autoscaled pool about newly added requests (#2400) (a90177d)

Features

3.8.2 (2024-03-21)

Note: Version bump only for package @crawlee/basic

3.8.1 (2024-02-22)

Note: Version bump only for package @crawlee/basic

3.8.0 (2024-02-21)

Bug Fixes

Features

  • accessing crawler state, key-value store and named datasets via crawling context (#2283) (58dd5fc)
  • adaptive playwright crawler (#2316) (8e4218a)

3.7.3 (2024-01-30)

Note: Version bump only for package @crawlee/basic

3.7.2 (2024-01-09)

Note: Version bump only for package @crawlee/basic

3.7.1 (2024-01-02)

Note: Version bump only for package @crawlee/basic

3.7.0 (2023-12-21)

Features

3.6.2 (2023-11-26)

Note: Version bump only for package @crawlee/basic

3.6.1 (2023-11-15)

Note: Version bump only for package @crawlee/basic

3.6.0 (2023-11-15)

Features

3.5.8 (2023-10-17)

Note: Version bump only for package @crawlee/basic

3.5.7 (2023-10-05)

Bug Fixes

  • add warning when we detect use of RL and RQ, but RQ is not provided explicitly (#2115) (6fb1c55), closes #1773
  • ensure the status message cannot stuck the crawler (#2114) (9034f08)
  • RQ request count is consistent after migration (#2116) (9ab8c18), closes #1855 #1855

3.5.6 (2023-10-04)

Note: Version bump only for package @crawlee/basic

3.5.5 (2023-10-02)

Bug Fixes

Features

3.5.4 (2023-09-11)

Features

  • remove side effect from the deprecated error context augmentation (#2069) (f9fb5c4)

3.5.3 (2023-08-31)

Bug Fixes

  • browser-pool: improve error handling when browser is not found (#2050) (282527f), closes #1459
  • clean up inProgress cache when delaying requests via sameDomainDelaySecs (#2045) (f63ccc0)
  • pin all internal dependencies (#2041) (d6f2b17), closes #2040
  • respect current config when creating implicit RequestQueue instance (845141d), closes #2043

Features

  • core: add default dataset helpers to BasicCrawler (#2057) (e2a7544)

3.5.2 (2023-08-21)

Note: Version bump only for package @crawlee/basic

3.5.1 (2023-08-16)

Features

  • exceeding maxSessionRotations calls failedRequestHandler (#2029) (b1cb108), closes #2028

3.5.0 (2023-07-31)

Features

3.4.2 (2023-07-19)

Bug Fixes

  • basic-crawler: limit internalTimeoutMillis in addition to requestHandlerTimeoutMillis (#1981) (8122622), closes #1766

Features

  • core: add RequestQueue.addRequestsBatched() that is non-blocking (#1996) (c85485d), closes #1995
  • retryOnBlocked detects blocked webpage (#1956) (766fa9b)

3.4.1 (2023-07-13)

Note: Version bump only for package @crawlee/basic

3.4.0 (2023-06-12)

Note: Version bump only for package @crawlee/basic

3.3.3 (2023-05-31)

Bug Fixes

  • set status message every 5 seconds and log it via debug level (#1918) (32aede6)

Features

  • core: add Request.maxRetries to allow overriding the maxRequestRetries (#1925) (c5592db)

3.3.2 (2023-05-11)

Bug Fixes

  • respect config object when creating SessionPool (#1881) (db069df)

Features

  • allow running single crawler instance multiple times (#1844) (9e6eb1e), closes #765
  • router: allow inline router definition (#1877) (2d241c9)

3.3.1 (2023-04-11)

Bug Fixes

  • start status message logger after the crawl actually starts (5d1df7a)
  • status message - total requests (#1842) (710f734)

3.3.0 (2023-03-09)

Features

  • add basic support for setStatusMessage (#1790) (c318980)
  • move the status message implementation to Crawlee, noop in storage (#1808) (99c3fdc)

3.2.2 (2023-02-08)

Note: Version bump only for package @crawlee/basic

3.2.1 (2023-02-07)

Note: Version bump only for package @crawlee/basic

3.2.0 (2023-02-07)

Bug Fixes

  • declare missing dependency on tslib (27e96c8), closes #1747

3.1.4 (2022-12-14)

Bug Fixes

3.1.3 (2022-12-07)

Bug Fixes

Features

  • always show error origin if inside the userland (#1677) (bbe9045)

3.1.2 (2022-11-15)

Note: Version bump only for package @crawlee/basic

3.1.1 (2022-11-07)

Note: Version bump only for package @crawlee/basic

3.1.0 (2022-10-13)

Note: Version bump only for package @crawlee/basic

3.0.4 (2022-08-22)

Note: Version bump only for package @crawlee/basic