Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore: do not fail when setting status message times out
c7488c4 Martin Adámek committed 3 years ago
fix: ensure the status message cannot stuck the crawler (#2114)
9034f08 Martin Adámek committed 3 years ago
chore(release): update internal dependencies [skip ci]
77600c7 Apify Release Bot committed 3 years ago
chore(deps): update styfle/cancel-workflow-action action to v0.12.0 (#2112)
aa7168f renovate[bot] committed 3 years ago
chore(deps): bump postcss from 8.4.30 to 8.4.31 in /website (#2113)
5cd59ae dependabot[bot] committed 3 years ago
docs: add links to the new deployment guides to the homepage
93cf279 Martin Adámek committed 3 years ago
feat: add incapsula iframe selector to the blocked list (#2111)
2b17d8a Jindřich Bär committed 3 years ago
fix(types): re-export RequestQueueOptions as an alias to RequestProviderOptions (#2109)
0900f76 Vlad Frangu committed 3 years ago
chore(release): update internal dependencies [skip ci]
6b681a4 Apify Release Bot committed 3 years ago
fix: allow to use any version of puppeteer or playwright (#2102)
0cafceb Martin Adámek committed 3 years ago
fix(templates): install browsers on postinstall for playwright (#2104)
323768b Vlad Frangu committed 3 years ago
fix(types): make return type of RequestProvider.open and RequestQueue(v2).open strict and accurate (#2096)
dfaddb9 Vlad Frangu committed 3 years ago
refactor: change `Initializing the crawler` log to `Starting the crawler`
654c4e7 Martin Adámek committed 3 years ago
refactor: use node loader instead of `ts-node-esm` in templates
52b47d2 Martin Adámek committed 3 years ago
Revert "feat: newUrlFunction receives Request and can return false"
27a0fe7 Jindřich Bär committed 3 years ago