Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
refactor: remove `additionalBlockedStatusCodes` parameter of `Session.retireOnBlockedStatusCodes`
577135d Martin Adámek committed 1 year ago
refactor: remove `additionalBlockedStatusCodes` parameter of `Session.retireOnBlockedStatusCodes`
9e8ad43 Martin Adámek committed 1 year ago
refactor: make crawling context strict and remove the error fallback
8ce049c Martin Adámek committed 1 year ago
chore(release): update internal dependencies [skip ci]
5dbdf0e Apify Release Bot committed 3 months ago
fix: respect `maxCrawlDepth` in `JSDOM/LinkeDOM` context `enqueueLinks` (#4021)
4d34aea Martin Adámek committed 3 months ago
fix(playwright): update handleCloudflareChallenge for new Cloudflare challenge markup (#4019)
379f629 Martin Adámek committed 3 months ago
chore: fix CI failures caused by lockfile maintenance (#4004)
15d95bd Martin Adámek committed 3 months ago
docs: move v3.17 recap after the v3.18 changes in the release post (#4003)
903fe69 Martin Adámek committed 3 months ago
chore(deps): update endbug/add-and-commit action to v11 (#4000)
a46420b renovate[bot] committed 3 months ago
docs(contributing): fix jest reference in vitest mock example (#3996)
783951d Bibek Barik committed 3 months ago
chore(deps): bump nanoid from 3.3.12 to 3.3.17 (#3994)
2f01d8c dependabot[bot] committed 3 months ago
chore(deps): bump js-yaml from 4.3.0 to 4.3.1 (#3995)
8b8b00b dependabot[bot] committed 3 months ago
chore(deps): bump mermaid from 11.15.0 to 11.16.1 in /website (#3993)
5dee23f dependabot[bot] committed 3 months ago
fix(core): do not purge storages that are already in use (#3988)
8e8c75b Jindřich Bär committed 3 months ago