Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix: Optimize request unlocking to get rid of unnecessary unlock calls (#2963)
a433037 Jan Buchar committed 1 year ago
fix(core): use short timeouts for periodic `KVS.setRecord` calls (#2962)
d31d90e Martin Adámek committed 1 year ago
fix(social): extract emails from each text node separately (#2952)
799afc1 Ondrej Klinovsky committed 1 year ago
chore(release): update internal dependencies [skip ci]
279cadb Apify Release Bot committed 1 year ago
fix: await `_createPageForBrowser` in browser pool (#2950)
27ba74b Martin Adámek committed 1 year ago
fix: respect `autoscaledPoolOptions.isTaskReadyFunction` option (#2948)
fe2d206 Martin Adámek committed 1 year ago
fix(statistics): track actual request.retryCount in Statistics (#2940)
c9f7f54 Lukáš Křivka committed 1 year ago
chore(deps): bump http-proxy-middleware from 2.0.7 to 2.0.9 in /website (#2938)
57f81b6 dependabot[bot] committed 1 year ago
chore: disallow yarn updates, allow docusaurus updates (#2935)
7250d81 Martin Adámek committed 1 year ago
feat: pass `response` to `FileDownload.streamHandler` (#2930)
008c4c7 Jindřich Bär committed 1 year ago
fix: disable default fingerprints in the Camoufox template (#2928)
6dffa00 Jindřich Bär committed 1 year ago
refactor: Minor adjustments to AdaptivePlaywrightCrawler (#2912)
a4fa838 Jan Buchar committed 1 year ago
chore(hotfix): Fix SitemapRequestList persist state event binding, vol. 2 (#2926)
6a1192a Jan Buchar committed 1 year ago
chore(hotfix): Fix SitemapRequestList persist state event binding (#2924)
a6a7073 Jan Buchar committed 1 year ago