Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore: try to fix extraction of client version from package.json (#1204)
904b847 Martin Adámek committed 5 years ago
fix: merge cookies provided by user with session cookies (#1201)
cccea78 Martin Adámek committed 5 years ago
fix: use encodeURIComponent instead of encodeURI (#1198)
cd98da0 Szymon Marczak committed 5 years ago
feat(SessionPool): allow passing `forceCloud` down to the KV store (#1186)
b86a4f2 Vlad Frangu committed 5 years ago
fix: allow passing `requestsFromUrl` to `RequestListOptions` in TS
c0ae5bd Martin Adámek committed 5 years ago
refactor: use `@apify/tsconfig` with enabled `esModulesInterop` (#1182)
a8feb28 Martin Adámek committed 5 years ago
feat: support channel and user links in YouTube regex (#1178)
65f2e91 Vlad Frangu committed 5 years ago
fix: automatically convert gdoc share urls to csv download ones in req. list (#1174)
ad79fee Martin Adámek committed 5 years ago
feat(InfiniteScroll): use puppeteer emulating scrolls instead of window.scrollBy (#1170)
a22b8aa Vlad Frangu committed 5 years ago
chore(deps): update dependency typescript to v4.4.3 (#1175)
ff4d4ba renovate[bot] committed 5 years ago
fix: fix `YOUTUBE_REGEX_STRING` being too greedy (#1171)
6c21c4a Martin Adámek committed 5 years ago