Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix(utils): parse sitemaps whose elements use a namespace prefix (#4393)
2587878 SwastikTripathi committed 6 days ago
fix(utils): ignore microdata inside <template> elements (#4400)
7416673 SwastikTripathi committed 6 days ago
fix(utils): skip <template> content in htmlToText (#4399)
1ef2a93 SwastikTripathi committed 6 days ago
chore(deps): update dependency puppeteer to v25.17.0 (#4392)
eb7b034 renovate[bot] committed 7 days ago
chore(deps): dependency sweep — in-range freshening (#4379)
393ecd7 apify-deps[bot] committed 8 days ago
fix(utils): extract URLs with internationalized top-level domains (#4380)
019689a SwastikTripathi committed 8 days ago
fix(utils): hash raw sitemap content once per source, not per url (#4381)
d2d2ad5 Robert Säll committed 8 days ago
fix: close the page of a timed-out adaptive browser attempt right away (#4371)
80ec501 Nikita Chapovskii committed 9 days ago
refactor!: remove the Google Sheets URL rewrite from `downloadListOfUrls` (#4377)
2648bc8 Jindřich Bär committed 9 days ago
feat(http): add an ignoreTlsErrors option to FileDownload (#4362)
1fa8d39 Vlad Frangu committed 10 days ago
chore: small code fixes found by the v4 upgrade skill audit (#4361)
74ab27f Vlad Frangu committed 10 days ago
chore: Remove the -Secs suffix from lock methods on RequestQueueBackend interface (#4276)
abddcfe Jan Buchar committed 10 days ago
fix(utils): separate HTML5 block elements and table headers in htmlToText (#4366)
589621f SwastikTripathi committed 10 days ago
chore(deps): update patch/minor dependencies to v1.65.0 (#4367)
58f479b renovate[bot] committed 10 days ago
chore(deps): dependency sweep — security bump + in-range freshening (#4368)
4d7207d apify-deps[bot] committed 10 days ago
chore(e2e): force-include copied packages when pushing test actors (#4365)
31467aa Martin Adámek committed 12 days ago
chore(deps): update dependency puppeteer to v25.14.0 (#4278)
f810b35 renovate[bot] committed 13 days ago
fix(basic): preserve request error messages without stacks (#4092)
8086839 Alperen committed 13 days ago
fix(puppeteer): keep intercept override headers passed as a `Headers` instance (#4351)
1be0209 Martin Adámek committed 13 days ago
chore(deps): dependency sweep — lockfile refresh (#4358)
640a7ba apify-deps[bot] committed 13 days ago
chore(deps): dependency sweep — lockfile refresh + esbuild floor fix (#4357)
410829b apify-deps[bot] committed 13 days ago
fix(basic): stop SitemapRequestLoader from looping on cyclic sitemap indexes (#4172)
ec5efcc breken committed 13 days ago
fix(basic): compile include/exclude patterns once per `addRequests` call (#4350)
6df2aa6 Luigi Ruocco committed 14 days ago
fix(core): stop draining async iterables after maxNewRequests is exhausted (#3579)
5e834e9 Junghwan committed 14 days ago
fix(utils): keep whitespace of elements nested in <pre> in htmlToText (#4348)
b04cf00 Wei-Lun Li committed 14 days ago
fix(basic): ignore completed domains when waiting for cooldowns (#4294)
09d3ffe Junghwan committed 14 days ago