Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore(deps): update actions/setup-node action to v4 (#2142)
ba449b2 renovate[bot] committed 3 years ago
chore(release): update internal dependencies [skip ci]
4378692 Apify Release Bot committed 3 years ago
fix(MemoryStorage): ignore invalid files for request queues (#2132)
fa58581 Vlad Frangu committed 3 years ago
chore: disable renovate updates on docusaurus-plugin-typedoc-api
3ccd4ac Martin Adámek committed 3 years ago
docs: downgrade docusaurus and typedoc to fix broken links in api docs (#2128)
5033339 Martin Adámek committed 3 years ago
fix: refactor `extractUrls` to split the text line by line first (#2122)
7265cd7 Martin Adámek committed 3 years ago
chore(release): update internal dependencies [skip ci]
da74c20 Apify Release Bot committed 3 years ago
fix: RQ request count is consistent after migration (#2116)
9ab8c18 Jindřich Bär committed 3 years ago
fix: add warning when we detect use of RL and RQ, but RQ is not provided explicitly (#2115)
6fb1c55 Martin Adámek committed 3 years ago